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Executive Summary 


Southwest Tennessee Community College (Southwest) received a Trade Adjustment 
Assistance Community College and Career Training (TAACCCT) grant from the U.S. 
Department of Labor in 2013 to fund the Southwest Solutions program (Solutions). 
The goal of Solutions is to prepare, credential, and place veterans, dislocated 
workers, and unemployed or underemployed individuals in high-skill occupations 
within the manufacturing process control industry. The college received the 
Solutions grant through the third round of TAACCCT funding. 


Industrial process control technology professionals are responsible for planning, 
analyzing, and controlling manufacturing production processes. The Solutions 
program awards a series of stacked and latticed credentials to students interested in 
or already employed in the field. Solutions includes a scale-up of Industrial 
Readiness Training (IRT), a manufacturing workforce readiness course, as well as a 
for-credit technical certificate and an associate of applied science (AAS) degree at 
Southwest in industrial process control technology (IPCT). Solutions also supported 
short-term training at manufacturing sites and at Southwest through the Lean Six 
Sigma Yellow Belt and International Organization for Standardization (ISO) Internal 
Auditor Training. 


Evaluation Design Summary 


CNA Education served as the third-party evaluator for the Solutions program. The 
evaluation focused on program implementation and students’ academic outcomes. 
The implementation evaluation used a qualitative case-study approach to gather 
feedback on the implementation of the program. This formative component of the 
evaluation sought to answer the four research questions listed below. Sub-questions 
1A through 1C were specified in the TAACCCT Round 3 request for proposals: 


1. How did the Solutions grant build institutional capacity at Southwest 
Tennessee Community College? 


A. How was the particular curriculum selected, used, or created? 
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B. How were programs and program design improved or expanded using 
grant funds? What delivery methods were offered? What was the program 
administrative structure? What support services and other services were 
offered? 


C. What contributions did each of the partners (employers, workforce system, 
other training providers and educators, philanthropic organizations, and 
others, as applicable) make toward: (1) program design, (2) curriculum 
development, (3) recruitment, (4) training, (5) placement, (6) program 
management, (7) leveraging of resources, and (8) commitment to program 
sustainability? What factors contributed to partners’ involvement or lack of 
involvement in the program? Which contributions from partners were most 
critical to the success of the grant program? Which contributions from 
partners had less of an impact? 


2. To what extent was grant implementation consistent with the original design? 
What were the institutional and external factors that contributed to changes in 
implementation? 


3. What are the most promising practices in the implementation of the Solutions 
grant? 


4. What were the challenges to successful implementation of the Solutions grant? 


CNA answered these questions using data collected from multiple sources, including 
interviews with grant staff and employer partners, surveys of students who 
participated in IRT and short-term training, and an analysis of program documents. 


The original design of the impact evaluation was to use a quasi-experimental 
approach to assess the effect of the for-credit IPCT certificate and AAS degree 
programs on students’ education and employment outcomes. This design was not 
feasible given that the for-credit programs were not approved until the spring of 
2016. There have been few participants in IPCT courses and no completers of 
certificates or AAS degrees. In addition, the college did not provide comparison 
group data for the for-credit participants. 


Given these limitations, the report includes descriptive statistics about Solutions 
participants and their academic outcomes. It summarizes the demographics of all 
students touched by the Solutions grant. These students include IRT participants, 
participants in other grant-funded, short-term training (Lean Six Sigma Yellow Belt 
and ISO Internal Auditor Training), and students enrolled in courses in another for- 
credit program, Advanced Integrated Industrial Technology (AIIT). Grant staff 
members anticipated that some AIIT courses would also count toward the for-credit 
programs in IPCT. In addition, AIIT students used equipment funded by the Solutions 
grant in their courses. Therefore, the AIIT students included in the report were 


ii 
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counted as grant participants. Outcome data for IRT participants include program 
completion rates and changes in scores on pre- and post-WorkKeys assessments. 
Finally, the report examines patterns of credits earned and year-to-year persistence 
among AIIT enrollees. 


Implementation Findings 


The Solutions grant led to the development of a stacked and latticed credential 
program in industrial process control technology. Southwest purchased additional 
equipment and took steps toward the implementation of technical certificate and 
AAS degree programs in the discipline. However, Southwest experienced 
implementation challenges that precluded the full implementation of all program 
components. These challenges included: delays in securing approval for the program 
from the Tennessee Board of Regents and equipment to support coursework, 
difficulties related to the organizational structure of grant administration, and 
limited staff capacity. 


The implementation evaluation results section of this report includes information on 
variations between the implemented program and the original program design, 
challenges to implementation, and implementation successes. It is organized by 
program component as follows. 


Stacked and latticed credentials: Southwest developed a stacked and _latticed 
credential sequence in industrial process control, which was fully offered to students 
in the spring of 2016, more than two years into the grant period. Students who 
complete IRT are eligible to enroll in for-credit technical certificate and AAS degree 
programs at Southwest in IPCT. With grant funding, the college purchased the 
instructional equipment necessary to support the program. A_ delayed 
implementation timeline prevented on-time, full implementation of this program 
component. In addition, the administrative structure of the grant led to confusion 
about who should take the lead on program implementation decisions. 


Employer engagement: The original Solutions proposal included a plan for an 
employer partner workgroup, which was not established. Southwest could have 
benefited from establishing stronger relationships with employer partners prior to 
grant implementation. In the final year of the grant, however, Southwest established 
a TAACCCT 3 Advisory Board to ensure that the college’s Round 3 TAACCCT grants 
were well-aligned with workforce needs. Moreover, Southwest participates in the 
Process Technology Talent Sector Council, which provides regional feedback on the 
extent to which industrial process control technology programs align with employer 
needs. The delayed implementation timeline and college staff turnover led to some 
challenges with employer engagement; yet industry partners provided positive 
feedback on the for-credit IPCT curriculum. 


ii 
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Program marketing and participant recruitment: The delayed implementation 
timeline led to difficulty recruiting prospective IPCT students, who were reluctant to 
enroll in a program that was not officially offered by the college. This challenge was 
compounded by the fact that prospective students often had a limited understanding 
of the industrial process control technology field and the career opportunities 
available within it. Grant staff members implemented new marketing and 
recruitment strategies in the final year of implementation. 


Prior learning assessments: In the original Solutions proposal, Southwest intended 
to support Learning Counts, an online portfolio for students developed by the 
Council for Experiential and Adult Learning (CAEL), which would have included prior 
learning assessments. Grant staff members determined in the spring of 2014 that the 
implementation of Learning Counts was not feasible and instead partnered with 
CAEL to develop challenge assessments for for-credit process control courses. 
Challenge assessments are locally specific tests that allow students to demonstrate 
prior learning related to a particular course. 


Data and tracking participants: Southwest intended to use Salesforce, a database 
software program, to track students’ progress through the process control career 
pathway. The software ultimately did not meet the Tennessee Board of Regent’s 
security requirements. Instead, Southwest purchased G*Stars, a program designed to 
track TAACCCT student data. It was not implemented until the fall of 2015. In the 
final year of grant implementation, grant staff members transitioned data and 
student case-management activities to other college offices to ensure the 
sustainability of data tracking and student support activities. 


Lessons Learned 


Based on the findings of the evaluation, including the challenges experienced during 
implementation, the report includes a series of lessons learned from the 
implementation of Solutions. These could be beneficial for other colleges considering 
the implementation of similar workforce development programs. 


e Consider the most appropriate design for the program. According to 
interview data, the college should have considered, prior to program 
implementation, whether the length of the for-credit programs aligned with the 
level of commitment that students would be willing to make to coursework. An 
alternative approach could have been to develop a short-term, non-credit 
training program in process control that led to a for-credit program. 


e Conduct a pre-implementation program planning process. The Solutions 
grant would have benefited from an in-depth program planning period prior to 
the submission of the grant proposal. A planning period would have allowed 


iv 
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grant staff members to make decisions related to curricular content, marketing 
and recruitment, and grant administration before the grant period started. 


e Engage with employers prior to and early in the implementation phase. An 
employer advisory board for TAACCCT 3 was not established until 2016, three 
years into the grant period. An earlier focus on employer engagement could 
have facilitated ongoing collection of employer feedback and supported 
additional recruitment efforts. 


e Establish clear lines of administrative authority and succession plans. The 
division of responsibility for grant administration between the workforce 
development department and the for-credit technologies department led to 
confusion over authority of the grant. Staff members should have established 
an organizational chart for grant administration, with clear lines of authority 
and responsibility and information on whom to contact in each department for 
specific types of information. The college would have also benefited from the 
development of a succession plan outlining steps for onboarding a staff 
member into a new role. 


Participant Outcomes 


Students who enrolled in courses or modules in another program, AIIT, were counted 
as grant participants. These students were enrolled in courses that Southwest 
anticipated would also count toward the IPCT credential. In addition, these courses 
used grant-funded equipment. This report includes descriptive results on students 
enrolled in for-credit AIIT courses touched by the grant, as well as IRT participants 
and short-term training participants supported through grant funds. 


In total, 407 students completed intake forms and were counted as Solutions 
participants. Of these, 176 participated in IRT, 107 participated in AIIT, 93 
participated in short-term training (Lean Six Sigma and ISO), and 31 were enrolled in 
another program at the college. The report includes disaggregated results for IRT, 
AIIT, short-term training, and other participants. 


Students enrolled in AIIT courses are approximately 36 years of age on average, while 
students in IRT are approximately 35. IRT students are less likely to be male than all 
Solutions participants, while AIIT students are almost all male. A larger percentage of 
IRT students are African American (92 percent) compared with AIIT students (49 
percent). AIIT students are also less likely to be employed at the time of enrollment 
than IRT students. ISO and Lean Six Sigma participants were older, on average, than 
IRT participants, and the majority of students were female. 
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IRT students take three WorkKeys assessments as part of the program to earn a 
National Career Readiness Certificate. IRT students are more likely to have identical 
WorkKeys scores at the beginning and the end of the program than to increase or 
decrease their scores. However, the next most likely outcome was that students 
improved their scores by one level. 


Among AIIT students, 71 percent of students who were enrolled during the first year 
of data collection persisted to the second year of data collection. These students 
tended to earn more credits during each specific year than students enrolled only 
during that year. 


e 
ANALYSIS & SOLUTIONS 


Contents 


Overview of the Southwest Solutions Program and Evaluation... 1 
The Southwest Solutions Program as Proposed and Implemented...........::cceeeeeees 2 
Implementation Study DeSign ..........cccccsccscseescseecesecssseeseeecseeseseeseeeesaeseseeeaesateneaesateneass 6 
Outcomes Study DeSign.......cccsscscssceecsssecsecseeessecsecsssecsecesseceesaseeseesaseeseeeeseeseeeaseetenes 9 
Implementation Study Findings. ..........:cssccssssssseessneeseesseesseeseeeseeeseesseeeseeeseeeseeeseesseeeeaeees 11 
Stacked and Latticed Credential Development and Implementation...............008 11 
Variation from Original Program DeSig0 ........:.c:csecssesseeeessseeseeesseeseeesseneneeeseneneens 13 
Challenges to Successful Implementation of Stacked and Latticed Credentials 
Bees ce dees cue e ccc sacs Pa coesta gests sterets fe tued Stee ca easesetat asus: ceucceasvevuctcacestevecstes tease <isteaesssessierss 13 
Strengthened Institutional Capacity ........cccceececcesseeeceeseeeeceeseeseceseeseeceseeseeseseneeees 16 
Employer EngageMent........ccccccscscsccsssscsscseeessceeeecsecaesecsecassecaecaesecaecaeseeaecaeseeecaseesaesasensaneas 19 
Variation from Original Program DeSi8N ........:.c:cssceessseeseesseeseeeeseeseeesseeeeeeseeeneaes 20 
Challenges to Employer Engagement ..........ccccccscscssesceecssseceecesseeeeceeeeeeeseeeeseesenensanees 20 
Employer Engagement Strengths .........cccscscsccscsccssecseesececeeceecseeeeecsseeseessseseesaseneaees 22 
Program Marketing and Participant RecruitMent............ccceccscsesseeeeseeeeeeeeeeeeeeseseneaeees 24 
Implementation Challenges for Recruitment ...........:ccccscscesceteeeeseeeeeeeeeeseeeeeeeeeeeeees 25 
Marketing Activities in the Final Grant Year .........:ccccccessscceetsescseeeeeeeeeeseeeeeenseeeaees 25 
Prior Learning ASSCSSMETIUS .......cccceeceesceeeececeeeceeeeeeseeeaeeeaeesaeesaeeseesseesaeeeaeeeaeeeeeeeaeeneseneeeaees 27 
Data and Tracking Participant ..........cccscscsscsssscsscseesesceseeeeecaeeeeeceeeesecaseeeaecesensesateneaeeas 27 
Definition: Of a Parvici pant siscssiss ivesesihias dieveseree wie ch cia ilhisetheledetiadh cenielescten 28 
Transition of Data Responsibilities to Other OffiC€S .........ceeeseeeeeeteeeeeeeeeeeeees 28 
Outcomes Study: Descriptive Re@SUItS ..........:csscsseseesseessesseesseeseeesseeseeeseeeseeeseeesenensnenses 30 
Demographic and Enrollment Statistics ..........ccccssscsccssseceeceeseceeceseeeeecaeseeseceeseseeeenensanees 30 
IRT WorkKeys Results and Program Completion Ratt .......:cccscsscsccsseeeeceseeeeeeseneesees 32 
IRT Certificate and Program Completion RateS.........ccssccscecscseesseeseeseeeseeseeeeees 36 
ATIT Credits and Persistence .......cccsscsccscsceccssssesecsecsssecseesssecseeaseeceeeseneceeasaesaeeasneseeeaseeeanes 36 
Conclusion: Lessons Learned from the Implementation of Solutions..................0 38 
Considerations for Program DeSiQN.........:csssscsccsseessesesesecseseeseceeseeseeeeceeeeeeecateeseneatenearens 38 
Pre-Implementation Program Planning. ..........cccccesesceccesseceecesseeeeeeeeeseeceeeeceeceseeseecesensaeees 39 
Earlier Engagement with EMployer..........cccccssescscssscescesseceeceeeeceeeaeeeeeecaseeeeeceseeeeeeasensaneas 40 


e 
ANALYSIS & SOLUTIONS 


Establishment of Lines of Grant Administrative Authority and Succession Plans41 


Sustainability Plan DevelopMe nt ...........ccccsssseeseeeseeeceseseeeeeeceeseesesaeseesecaneeesesaseneeeateneaeees 41 
Implementation SUMMALY.......:ccccccsssscsscsssecesceseecseceeseeecaeseesecaesecaecaeeeeaecateeeaecaeeeseesanensaneas 42 
Appendix: Example of the Interview Protoco .........::::ccscseseseesseeseeeseeesseeseeesseeseneneeenaees 44 
ROTOR CM COS oe aei vices ccbevenciad sheds caccvenscaccyeusaddevctvacsyoctaucevasbedsascetedeasdeutesusuetkesescetesuretedeasdstedeuvanteds 49 


viii 


e 
ANALYSIS & SOLUTIONS 


List of Tables 


Table 1. Implementation Data-Collection ACtivitieS .........:cccceccsecsecesseeeceseeeeeceeeeseeesenenees 8 
Table 2. Implementation Timeline for Stacked and Latticed Credentials ..............000 12 
Table 3. Implementation Timeline for Employer Engagement ..........:cccssscseeeeeeesseeeeeees 20 
Table 4. Implementation Timeline for Recruitment Activities ..........cccceeeseeeeeeeeeeeeees 24 
Table 5. Implementation Timeline for Activities Related to Data and Tracking 
SOULGOMUS i sscaciss exe stivedsteccet teesucsecssseascusveerstearcdas aoneatce nats eeeaeeineaenaas 27 
Table 6. Demographic and Enrollment Characteristics of Solutions Participants........ 31 
Table 7. Level Changes in WorkKeys Applied Math Pre-Test/Post-Test Scores ...........: 33 
Table 8. Summary of Level Changes in WorkKeys Applied Math Pre-Test/Post-Test 
SCOLTES aisvedeceries axcceses te cevtes eeeeecevcedevecseneee, esta tectesteeratvn edicts seine eenscstectans se eeeeete ses 34 
Table 9. Level Changes in WorkKeys Locating Information Pre-Test/Post-Test 
SCOPES -isee secevses seeeeshs sutevtes eeeperen eeaveesevee seca rea teaai vie eeeniet ne ese tener 34 
Table 10. Summary of Level Changes in WorkKeys Locating Information Pre- 
TESt/POSt- TESt S COLES: viveveievissiassseswevsvacudsusestsnvieus coiaatasbivessanstaiateitentiavaesaaceusis 34 
Table 11. Level Changes in WorkKeys Reading for Information Pre-Test/Post-Test 
SCOPES cists sacccies deecesthtucevers seep seen eedscnses vere cateatevans te enefiesscz eters asesteraceeeeeeaeetines 35 
Table 12. Summary of Level Changes in WorkKeys Reading for Information Pre- 
TESt/POSt- TESE SCOPES: sissicsaccesisssseienvswacudencastens cea vvedactaabivaniaactansteitivvsnviasdaennaas 35 
Table 13. Level Changes in National Career Readiness Pre-Test/Post-Test Scores......35 


Table 14. Summary of Level Changes in National Career Readiness Pre-Test/Post- 


Te@St SCOLES wocscccccscccssscccseeeseesscccesecesscenescnesaeeseneeesnesaeeeesenesaneeaeaeeusnesananeeeneranesas 36 


ix 


e 
ANALYSIS & SOLUTIONS 


Glossary 


AAS Associate of Applied Science 

AIIT Advanced Integrated Industrial Technology 

CAEL Council for Adult and Experiential Learning 

GMACW Greater Memphis Alliance for a Competitive Workforce 

IPCT Industrial Process Control Technology 

IRT Industrial Readiness Training 

ISA International Society of Automation 

ISO International Organization for Standardization 

NCRC National Career Readiness Certificate 

NSCC Nashville State Community College 

OSHA Occupational Safety and Health Administration 

PLA Prior Learning Assessment 

SOCMA Society for Chemical Manufacturers & Affiliates 

TAA Trade Adjustment Assistance 

TAACCCT Trade Adjustment Assistance Community College 
Training 

TBR Tennessee Board of Regents 

WIN Workforce Investment Network 


e 
ANALYSIS & SOLUTIONS 


Overview of the Southwest Solutions 
Program and Evaluation 


CNA Education served as the third-party evaluator for the Southwest Solutions 
initiative (Solutions), which is funded through a Trade Adjustment Assistance 
Community College and Career Training (TAACCCT) grant from the U.S. Department 
of Labor. TAACCCT grants provide support to community colleges to develop, 
expand, and strengthen career education programs that can be completed in two 
years or less. Southwest Tennessee Community College (Southwest) received funding 
for the Solutions program through the third round of TAACCCT grant awards. The 
funding period began in October 2013 and continued through September 30, 2017. 


Southwest Solutions includes a series of stacked and latticed credentials for students 
interested in or already employed in the field of industrial process control 
technology (IPCT). Stacked and latticed credentials help workers advance along a 
career pathway, move up a career ladder (stacked), or shift to a related field 
(latticed). 


Industrial process control technology professionals are responsible for planning, 
analyzing, and controlling production processes in a variety of industries. Key skills 
for process control technicians include: measuring and feeding raw materials into 
machinery, setting and monitoring indicators, and testing and monitoring products 
moving through the manufacturing cycle. In Memphis and elsewhere, there is a need 
for process control technicians in oil and chemical refinement, food and beverage 
production, waste water treatment, and pulp and paper production.’ 


The Solutions grant program consisted of: a scale-up of Industrial Readiness Training 
(IRT), a four-and-a-half-week non-credit program designed to prepare students for 
entry-level industrial jobs; a year-long for-credit technical certificate; and an 
associate of applied science (AAS) degree program in IPCT. The program sought to 
serve 550 unique participants through IRT and for-credit programs, with 415 


' Information on the industries in which process control technology careers are available is 
from the website of the Nashville State Community College program in Industrial Process 
Control Technology. 
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participants completing a TAACCCT-funded program of study. The original program 
design emphasized collaborative partnerships with industry, community-based 
organizations, and other higher-education institutions; technology-enabled learning; 
prior learning assessments; and implementation of a new data system to track 
student progress and outcomes. 


Due to delays in the project timeline, challenges related to the grant and college’s 
organizational structure, and limits to staff capacity and capabilities, components of 
the Solutions program were not implemented as intended. In total, the grant-funded 
Solutions programs served 407 participants, 269 of whom participated in a noncredit 
TAACCCT-funded program (176 IRT students and 93 participants in two-day, short- 
term manufacturing training programs at Southwest). 


Because the for-credit IPCT programs were not offered until the spring of 2016, only 
5 students took IPCT courses by the spring of 2017. However, another 107 students 
enrolled in a different for-credit program, Advanced Integrated Industrial Technology 
(AIIT), were counted as participants. These students were also considered 
participants because grant staff members anticipated that AIIT courses would count 
toward IPCT credentials and because AIIT courses used grant-funded equipment. 


Despite the small number of IPCT enrollments, the grant increased Southwest’s 
institutional capacity through development of new for-credit programs, development 
of related short-term training programs, purchase of new instructional equipment, 
and stronger relationships with process control employers. The remainder of this 
section provides an overview of the Southwest Solutions program as it was proposed 
and implemented and an overview of the third-party evaluation design. 


The Southwest Solutions Program as Proposed 
and Implemented 


As proposed, the Solutions career pathway began with IRT, a four-and-a-half-week 
foundational program in industrial skills. Southwest and the Workforce Investment 
Network (WIN) developed IRT in 2012 in response to industry feedback that 
employers were having difficulty hiring candidates for entry-level manufacturing 
positions. Generally offered in partnership with an employer interested in hiring new 
workers, IRT provides students with the technical, academic, and workforce 
readiness skills necessary for jobs in the manufacturing sector. As part of IRT, 
students take a technical skills assessment developed by Southwest, as well as ACT 
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WorkKeys assessments in Applied Mathematics, Locating Information, and Reading 
for Information.’ 


Students who complete IRT earn a certificate of IRT completion, a National Career 
Readiness Certificate (NCRC) from ACT based on WorkKeys scores, and an 
Occupational Safety and Health Administration (OSHA) 10-hour General Industry 
Safety Certificate. Through the OSHA-related instruction, IRT provides participants 
with an awareness of workplace health and safety issues. IRT completers are 
considered prepared for entry-level employment and are eligible to transition into 
other technical training programs at Southwest, including the for-credit IPCT 
programs offered through Solutions. 


To continue the career pathway, Southwest adopted and implemented a for-credit 
technical certificate, Process Control Fundamentals, and an AAS degree in IPCT. The 
AAS degree has two concentrations: process control technician and maintenance 
technician. These technicians play an active role in the production process, from the 
acquisition of raw materials through the distribution of products to customers. 


Each Round 3 TAACCCT grantee was required to indicate how its project would 
address six core elements. The implemented Solutions program diverged from the 
proposal, largely due to a slow implementation timeline and limited staff capacity. 
Following the brief overview here, the implementation study findings provide greater 
detail on how the implemented program differed from the intended design and how 
implementation challenges contributed to these differences. 


Evidence-based design: The development and scale-up of the Solutions program was 
based on the design and evidence emerging from several stacked and latticed 
credential models, including the Integrated Basic Education and Skills Training (I- 
BEST) program in Washington State (Jenkins, Zeidenberg & Kienzl, 2009) and the 
Joyce Foundation’s Shifting Gears initiative (Roberts & Price, 2012). 


Stacked and latticed credentials: The Solutions proposal envisioned development of 
a stacked and latticed credential program in which IRT completers could transition 
to for-credit technical and AAS degree programs in IPCT. Solutions sought for all 
courses to align with the AAS degree, which was based on a program previously 
implemented at Nashville State Community College (NSCC). Following the selection of 
NSCC’s curriculum, Southwest was required to seek program approval from the 
Tennessee Board of Regents (TBR). The approval process took more than a year, 
which delayed the implementation. Process control technology courses were not 


? In the summer of 2017, ACT changed the names of the WorkKeys assessments to Applied 
Math, Graphic Literacy, and Workplace Documents; we refer to these assessments by their 
initial names, which would have applied for the vast majority of students. 
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offered to Southwest students until the spring of 2016. Due to the delayed timeline, 
Solutions faced challenges related to both recruitment and development of strong 
employer partnerships. 


Solutions funded the scale-up of the IRT program. Two other Round 3 TAACCCT 
grants at Southwest also supported the expansion of IRT. Which TAACCCT grant 
funded a specific section of IRT depended on the workforce needs of the employer 
sponsoring the class. Solutions funded sections of IRT when the sponsoring 
employer needed process control technology employees. 


In addition to IRT, Solutions staff members used grant funding to partner with 
process control employers to develop onsite training courses customized to their 
specific needs. In the final year of the grant, Southwest implemented additional 
short-term manufacturing training programs that provided students with certificates 
from the International Organization for Standardization (ISO) and Lean Six Sigma 
(Yellow Belt). Solutions, in addition to the other Round 3 TAACCCT grants awarded 
to Southwest, contributed to the cost of these training programs. 


Southwest also incorporated prior learning assessments (PLAs) into the Solutions 
proposal. Originally, Southwest planned to partner with Council for Adult and 
Experiential Learning (CAEL) to develop Learning Counts portfolios for participants 
to track PLA credit and academic progress. Grant staff members determined that it 
would not be possible to offer Learning Counts in the spring of 2014. Therefore, 
CAEL developed a series of PLAs, called challenge assessments, for specific courses, 
so that students could earn credit for previous training or experience. 


Transferability and articulation of credit: Technical certificate and AAS degree 
programs were based on an existing program at NSCC. TBR policy requires that 
programs offered in the same subject be consistent across the state of Tennessee. 
Thus, a student who began the program at Southwest could transfer credits to NSCC. 


The developers of Solutions originally proposed articulation agreements with four- 
year engineering programs at the University of Memphis and Christian Brothers 
University. Despite outreach efforts, these agreements were not established. Efforts 
are ongoing to establish articulation agreements with applied four-year degree 
programs at the University of Arkansas-Fort Smith and the University of Tennessee- 
Martin that align with IPCT courses. 


Advanced online and technology-enabled learning: The Solutions team proposed 
the adoption of an online portfolio that students could use to track their progress; it 
would use CAEL’s Learning Counts Portfolio as a platform. The platform would 
include results from KeyTrain, the career readiness training system used in IRT, and 
Amatrol, the instructional program used in the for-credit programs. Amatrol includes 
self-paced instruction online and via simulations with realistic manufacturing 
machinery in the Solutions laboratory space at Southwest. The portfolio would be 
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accompanied by a back-end data system allowing faculty and staff members to track 
progress through KeyTrain, Amatrol, and Learning Counts. Faculty would use 
Salesforce, a database software, to track student outcomes. 


The portfolio and data system were not implemented as intended. Students do not 
have access to an online tool to chart their progress, although they can still chart 
their progress in IRT through KeyTrain and in for-credit coursework through 
Amatrol. In the spring of 2015, the Solutions team determined that it was unable to 
purchase Salesforce due to the TBR security requirements. Therefore, Southwest 
purchased and implemented G*Stars, an online data tracking system designed 
specifically to track TAACCCT data. 


Strategic alignment with relevant community partners, especially businesses: Per 
the grant proposal, Southwest planned to establish five partner workgroups to 
facilitate the implementation of Solutions: (1) data collection, (2) online technology, 
(3) workforce development, (4) articulation agreements, and (5) employer partners. 
Southwest Solutions staff members engaged in efforts to develop these groups early 
in the grant period, but they were not established. 


Despite incomplete partnership-building, Southwest staff members worked to engage 
employers with the grant. In 2016, three years into the grant period, Southwest 
established a TAACCCT 3 Advisory Board for the three Round 3 grants on campus. 
The board provides guidance to Southwest to ensure that the college’s workforce 
development and industrial training programs are well-aligned with the needs of 
local industry. In addition, in partnership with the Greater Memphis Alliance for a 
Competitive Workforce (GMACW) and the Arkansas State University Mid-South, 
Southwest participates in the Process Technology Talent Sector Council. 
Postsecondary institutions and employers share information about industry needs 
and educational programming available. 


Alignment with previously-funded TAACCCT projects: Southwest partnered with 
Mount Wachusett Community College in Massachusetts, which received a Round 1 
TAACCCT grant that facilitated the development of business and information 
technology and reduced barriers to entry for biotechnology students. Throughout the 
grant period, Southwest stayed in contact with Mount Wachusett to share promising 
practices. In the fall of 2016, the Mount Waschusett connected Southwest with a 
workforce development consultant, who partnered with Southwest to develop a 
strategic plan for the remaining months of implementation. 


Design of the Solutions External Evaluation 


CNA Education served as the third-party evaluator for the Solutions program. The 
original intent was for the evaluation to use a mixed-methods research design with a 
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focus on the implementation and overall effectiveness of the Solutions initiative. 
However, the study design varied from the original design due to data limitations. 
This section describes the data and methodological approaches used to determine 
the results included in the final report. 


Implementation Study Design 


To guide the evaluation of Solutions, CNA developed a logic model in consultation 
with grant staff in 2014. It outlined the grant activities supporting six core 
components of the Solutions program and provided a framework for the 
development of the analysis plan (Figure 1). In the figure, the bolded items indicate 
grant activities that were implemented during the four-year performance period. 


Figure 1. Southwest Solutions Logic Model 


Outputs IRT 
* NCRC Four-and-a-half-week curriculum 
p LearningCounts 

Doha Online student dashboard 


Entry-level 
employment 


Technical Certificate 
Prior learning assessment 
12-month curriculum 
LearningCounts 


Entrepreneurial Online student dashboard 
certificate 


Beginner-level 
technician 
employment 


Outputs 
Technical 
certificate 


Outputs HRISUEENE of Applied Science (AAS) 
AAS degree Degree in EEE Control Technology 
ISA Certified Control *  24-month curriculum 

¢ LearningCounts 


systems Teehnician Online student dashboard 


certification 
Articulation to a 4- 
year degree 
Mid-level technician 
employment 


4-year degree 


e 
ANALYSIS & SOLUTIONS 


The implementation evaluation used a qualitative case study approach (Yin, 2014), 
gathering and triangulating feedback from multiple sources to assess the strengths 
and weaknesses of implementation. The implementation evaluation sought to answer 
several related research questions. Questions 1A - 1C were required implementation 
research questions within the Department of Labor solicitation for grant applications 
for TAACCCT Round 3. CNA included these questions under a broader question 
focused on how the Solutions grant built institutional capacity (question 1). The 
research questions for the implementation evaluation were: 


1. How did the Solutions grant build institutional capacity at Southwest 
Tennessee Community College? 


A. How was the particular curriculum selected, used, or created? 


B. How were programs and program design improved or expanded using 
grant funds? What delivery methods were offered? What was the 
administrative structure? What support services and other services were 
offered? 


C. What contributions did each of the partners (employers, workforce system, 
other training providers and educators, philanthropic organizations, and 
others as applicable) make toward: (1) program design, (2) curriculum 
development, (3) recruitment, (4) training, (5) placement, (6) program 
management, (7) leveraging of resources, and (8) commitment to program 
sustainability? What factors contributed to partners’ involvement or lack of 
involvement in the program? Which contributions from partners were most 
critical to the success of the grant program? Which contributions from 
partners had less of an impact? 


2. To what extent was grant implementation consistent with the original design? 
What were the institutional and external factors that contributed to changes in 
implementation? 


3. What are the most promising practices in the implementation of the Solutions 
grant? 


4. What were the challenges to successful implementation of the Solutions grant? 


To address these questions, CNA employed multiple data-collection strategies 
throughout the grant period (Table 1). We conducted annual site visits to Southwest 
to conduct in-depth interviews with grant staff members and tour laboratory 
facilities. During the 2017 site visit, CNA researchers observed a meeting of the 
TAACCCT 3 Advisory Board. In the fall of 2016 and spring of 2017, CNA conducted 
in-depth interviews with employer partners who either had a role in the development 
of the for-credit process control curriculum or collaborated with Southwest to 
implement customized training programs. The number of employer interviews was 
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limited to three due to other respondents’ lack of responsiveness or availability. 
Finally, CNA conducted an analysis of program documents during the last year of 
grant implementation. 


During Year 3, CNA surveyed participants who completed IRT or customized short- 
term training opportunities. The surveys focused on students’ reasons for enrolling, 
the extent to which programs improved students’ skills, and student satisfaction. 
Survey results are included in the 2016 interim evaluation report. 


Table 1. Implementation Data-Collection Activities 


Year Data-Collection Activities Participants 

2015 Semi-structured site visit Grant counselor, 
interviews with 8 grant staff employment specialists 
members (2), IRT instructors (2), in- 


take specialist, for-credit 
technologies instructor, 
and Solutions laboratory 


technician 
2016 Site visit interviews with 6 grant Grant counselor, 
staff members academic specialist, 


employment specialist, 
intake specialist, 
laboratory technician, 
and for-credit 
technologies instructor 


Survey of IRT and short-term Solutions-sponsored IRT 

training participants participants and 
completers of short-term 
training at Chemours and 
Steel Warehouse/SFI 


2017 Site visit interviews with 11 grant Associate Vice President 
staff members of Workforce 

Development, Director of 
Externally Funded 
Programs, grant 
coordinator, academic 
specialist, data 
administrator, IRT 
manager, IPCT 
coordinator, laboratory 
technician, career 
services director, student 
services director, and 
TAACCCT 3 consultant 


Employer partner interviews 3 employer partners 
Program document analysis 
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Interviews with staff members focused on program design and development; 
curriculum; student support services; student outcomes; successes; and 
programmatic challenges, lessons learned, and implications for sustainability. The 
Appendix includes an example of an interview protocol developed for these site 
visits. To analyze interview data, the evaluation team transcribed the interviews and 
coded them inductively, carefully reading the interview transcripts and developing 
descriptive codes designating themes related to implementation challenges, 
promising practices, and grant sustainability. 


The 2017 document analysis included quarterly reports that Southwest submitted to 
the U.S. Department of Labor, institutional documents related to program 
implementation, and Solutions promotional materials. CNA also analyzed documents 
related to the TAACCCT 3 Action Plan, a strategic plan for the final nine months of 
program implementation jointly developed by grant staff and an external consultant. 
The evaluation team coded the documents inductively, using the same process as for 
interview transcriptions. 


Outcomes Study Design 


The original outcomes evaluation research design proposed using a quasi- 
experimental approach to assess the impact of Solutions on participating students’ 
education and employment outcomes. However, the evaluation team was unable to 
conduct the outcomes evaluation as intended due to the small number of for-credit 
participants (e.g., no treatment group data for IPCT participants), the lack of 
comparison group data, and limited data on employment outcomes. This section 
describes the proposed study design, why it was not feasible, and an overview of the 
descriptive results provided in the evaluation report. 


Proposed Outcomes Evaluation Research Design 


As proposed, the evaluation sought to answer the following research questions: 


1. Do Southwest students participating in the Solutions program achieve better 
educational outcomes than non-participating Southwest students? 


2. Do Southwest students participating in the Solutions program achieve better 
employment outcomes than non-participating students? 


The evaluation proposed to use a propensity score matching design to compare 
outcomes for for-credit Solutions participants and a comparison group of students in 
similar for-credit programs at Southwest. CNA planned to track educational and 
employment outcomes. Educational outcomes included year-to-year retention in the 
program, credit accumulation, completion of a program of study, attainment of 
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industry-recognized credentials, and enrollment in further education. CNA also 
planned to track students’ employment and earnings outcomes during the program. 
The original design did not include a comparison group for IRT participants because 
there was no comparable population from which to create one. 


Infeasibility of Quasi-Experimental Design 


It was not feasible to implement propensity score matching because it was not 
possible to develop treatment or comparison groups for IPCT participants at 
Southwest. The for-credit technical certificate and AAS degrees were not available 
until the spring 2016 semester. No students have completed either program, and 
there were only 5 enrollments IPCT courses through the spring 2017 semester. In 
addition, the college did not provide comparison group data, a requirement of the 
propensity matching approach. 


The design was also complicated by for-credit students whom Southwest counted as 
grant participants before the launch of the IPCT courses. During the initial months of 
the grant, the college anticipated that the certificate and AAS degrees would share 
courses with another technology program, AIIT, which also used grant-funded 
equipment. Students enrolled in these courses were asked to fill out grant intake 
forms. Therefore, many students counted as participants were likely pursuing a 
credential unrelated to the Solutions TAACCCT grant. It is difficult to distinguish 
which students, if any, intended to earn a credential in IPCT. 


Descriptive Results on Solutions Participants Included in the Report 


The results include descriptive statistics on all students who completed intake forms 
and were counted as Solutions participants, as well as disaggregated data on 
participants in IRT, AIIT, short-term training supported by the grant, and other 
programs. In addition, the report includes descriptive information on medium-term 
outcomes for IRT and AIIT participants. Results on IRT participants include 
comparisons of WorkKeys test scores at the beginning and the end of IRT and their 
program completion rates. The AIIT results focus on credit attainment and 
persistence. 
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Implementation Study Findings 


This section presents the final results of the implementation evaluation, based on the 
final round of interviews with grant staff members, interviews with employers, and 
an analysis of program documents. The findings are organized by program 
component, as follows: 


e Stacked and latticed credential development and implementation 
e Employer engagement 
e Marketing and participant recruitment 
e Prior learning assessments 
e Data and tracking participants. 
Each subsection includes: 
e An overview and timeline of program implementation 
e A discussion of how implementation varied from the original program design 


e A description of challenges related to the successful implementation of the 
grant 


e A description of promising practices and how grant activities in each area 
increased Southwest’s institutional capacity. 


Stacked and Latticed Credential 
Development and Implementation 


The Solutions grant program was designed to be a stacked and latticed credential 
program within the field of process control. The career pathway began with a scale- 
up of IRT, an existing manufacturing readiness training program that provides 
participants with foundational technical, academic, and workforce readiness skills. 
IRT prepares students for both entry-level manufacturing positions and to articulate 
into for-credit technologies programs at Southwest. 
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Through the Solutions grant, Southwest adopted and implemented for-credit IPCT 
programs, including a 23-credit process control technology certificate and a 60-credit 
AAS degree with maintenance technician and process control technician 
concentrations. The Solutions grant proposal envisioned that the for-credit program 
would be implemented during the 2014/15 academic year, with 40 students 
completing credit hours in Year 1. In reality, the program approval process took 
much longer than anticipated and delayed the implementation timeline (Table 2). 


Table 2. Implementation Timeline for Stacked and Latticed Credentials 


Date Solutions Project Milestone 


Fall 2014 Contracted with a curriculum specialist to 
develop a plan for the technical certificate 
and AAS, based on an existing curriculum 
at Nashville State Community College 


December 2014 Draft curriculum completed 
List of equipment to purchase submitted to 
the U.S. Department of Labor 


Spring 2015 Abstract of technical certificate and AAS 
curricula submitted to the TBR 

Spring 2015 — Present Discussion of potential national certificates 
to incorporate into the certificate and AAS 
degree 

Summer 2015 Program abstract approved by the TBR, 


which proceeded with an additional 30- 
day review process 


November 2015 TBR approval of the certificate and AAS 
degree in IPCT 


2015 — 2016 academic year Development of corporate training courses 
on process control topics; corporate 
training at Steel Warehouse/SFI and 


Chemours 

Fall 2016 Additional corporate training at Steel 
Warehouse/SFI 

Spring 2017 Development and implementation of short- 


term job training programs from the ISO and 
Lean Six Sigma 
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Variation from Original Program Design 


By the conclusion of the four-year grant period, the college had implemented a career 
pathway program for Solutions that included IRT, additional short-term training 
related to industrial careers, and the IPCT technical certificate and AAS degree. 


IRT was originally developed as part of the Made in Memphis initiative, a partnership 
between Southwest, the Workforce Investment Network (WIN), and the Greater 
Memphis Chamber of Commerce designed to create a pipeline of skilled workers to 
meet the needs of local manufacturing. Participants who complete IRT receive a 
certificate of IRT completion, an ACT National Career Readiness Certificate, and an 
Occupational Safety and Health Administration (OSHA) 10-hour General Industry 
Safety Certificate. The three Round 3 TAACCCT grants at Southwest supported 
sections of IRT based on specific companies’ training needs. If a company needed to 
fill process control-related jobs, Solutions funded the section of IRT. 


Staff members also invested grant funds in short-term workforce training 
customized to employer needs. Southwest developed a syllabus and course outline 
for corporate training materials used to teach process control at local employers’ 
facilities. Faculty members partnered with Chemours and Steel Warehouse/SFI to 
offer customized process control training at the companies’ manufacturing work 
sites. In addition, Solutions funds supported the implementation of free certification 
classes for ISO Internal Auditor Training and Lean Six Sigma Yellow Belt. 


The 2013 Solutions grant proposal included plans for incorporating an 
entrepreneurial certificate into the technical certificate for students interested in 
starting their own businesses, which would have been developed in partnership with 
the Kaufman Foundation’s JumpStart program. In addition, program designers 
intended that the AAS degree lead to an industry-recognized certificate, the 
International Society of Automation (ISA) Certified Controls Systems Technician. 
Neither the Jumpstart nor the ISA programs were implemented, largely because 
approval of the for-credit programs took much longer than anticipated. At 
Southwest, discussion of incorporating national industry-recognized certifications 
into the technical certificate and AAS degrees continues. 


Challenges to Successful Implementation of Stacked 
and Latticed Credentials 


During the grant period, several challenges slowed or inhibited the full 
implementation of the for-credit technical certificate and AAS degree. Primary 
obstacles included: a delayed timeline, the organizational structure of grant 
administration at Southwest and related staff capacity limitations, and grant staff 
and leadership turnover. 


13 


e 
ANALYSIS & SOLUTIONS 


Delayed Implementation Timeline 


The delayed timeline for launching the for-credit programs was a major challenge for 
implementing the Solutions program as intended. Although those who designed the 
program anticipated offering the process control curriculum during the 2014/15 
academic year, Southwest did not offer IPCT as an official program until the spring 
2016 semester. In addition, procuring equipment using grant funds took longer than 
anticipated. 


The original grant proposal did not include time for selection, development, and 
approval of the draft IPCT curriculum. The first grant coordinator was hired in the 
spring of 2014. Following his hire, startup activities took several months. In the fall 
of 2014, Southwest contracted with subject matter experts to develop the 
curriculum. During development, employer partners reviewed and commented on the 
curriculum content. An abstract of the draft curriculum was submitted to the TBR in 
early 2015. 


It is unclear why it took several months for the TBR to approve the curriculum 
abstract. The college received an initial, tentative approval in July 2015. At that time, 
the TBR initiated another 30-day approval process, which also took longer than 
anticipated. E-mail correspondence between the grant coordinator and a contact at 
the TBR indicates that there was some confusion about the level of detail required in 
the program application, given that it was a near duplication of an existing program 
at NSCC. Due to these delays, the college did not receive full approval from the TBR 
to offer the for-credit programs until November 2015. 


The delay in the approval process contributed to challenges with student recruitment 
and the development of employer partnerships, described in subsequent sections of 
this report. 


Procurement of hands-on learning equipment for the Solutions laboratory at 
Southwest was similarly delayed. Grant staff considered the process administratively 
burdensome. The grant coordinator submitted a purchase requisition to the U.S. 
Department of Labor in late 2014 but did not receive approval because the document 
was incomplete. Based on feedback from the project officer at the Department of 
Labor, the grant’s academic specialist revised and resubmitted the document but did 
not receive a response. Six months later, the college learned that it had submitted the 
request in an incorrect format. Grant staff members said that they would have liked 
additional and quicker feedback from the Department of Labor so that they could 
have completed the equipment procurement process more efficiently. 


Challenges with the Organizational Structure of the Grant 


The organizational structure of the grant administration team may have also 
contributed to the delayed implementation timeline. The grant originally funded both 
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a grant coordinator and a finance and procurement specialist. From the beginning of 
the grant period until August 2016, roles were designated so that the procurement 
specialist served as the sole liaison between Southwest and the Department of Labor. 
She initiated all communication between the two organizations. The grant 
coordinator chose not to communicate directly with the project officer at the 
Department of Labor, which likely slowed the equipment procurement process. 


The grant proposal’s original timeline did not account for start-up and planning time 
required prior to implementing grant activities. After receiving the TAACCCT grant, 
Southwest hired all new staff with roles specific to the grant, some of whom had 
little experience in community college workforce development programs. According 
to one interview respondent, at the beginning of the grant period, “it was hard to just 
bring new people in and expect them to know exactly what to do...everybody was 
new and trying to get their feet under them.” A grant staff member explained that 
the workforce development department’s leadership had not expected to win all 
three Round 3 TAACCCT grants and did not have sufficient organizational capacity 
to effectively implement the three projects simultaneously. 


In addition, the grant required communication and coordination across and within 
college departments. Some interdepartmental confusion resulted from the fact that 
the grant was administered through the non-credit workforce development 
department, but the IPCT instructional programming would be implemented by the 
for-credit technologies department. There was confusion about who should make 
decisions related to the grant. When interviewed in 2017, grant leadership indicated 
that the original grant coordinator did not feel empowered to provide direction to 
academic faculty or make decisions related to the grant without consulting faculty 
members. The grant coordinators also experienced challenges navigating the 
college’s bureaucracy— for example, a lack of responsiveness from departments such 
as accounts payable that were not funded directly by the grant. These departments 
were often unfamiliar with the grant or their role in supporting it. According to one 
of the coordinators, “things would have been a lot easier if every time you went to 
interact with someone on behalf of your grant, you didn’t have to go into the 
conversation about who are you and what are you doing.” 


Turnover 


Ongoing turnover of grant staff, as well as departmental and college leadership, was 
a major challenge for the implementation of the Solutions grant. There have been 
three leadership transitions in the workforce development department, including one 
currently ongoing, over the course of the grant. The most recent Associate Vice 
President of Workforce Development assumed her position in August 2016 and 
served until June of 2017. Each change in management required time to reorganize 
and strengthen the grant team. 
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The grant’s staffing structure changed several times during implementation, which 
led to a loss of institutional memory when staff members transitioned off the 
project. There were three grant coordinators during the grant period. The second 
grant coordinator was familiar with the project, since she had previously served as a 
grant counselor. Nonetheless, each transition required time for the new coordinator 
to become familiar with his or her new role. 


Other staff members turned over as well. At first, the grant supported guidance 
counselors and employment specialists, who were tasked with building and 
strengthening relationships with employer partners. The employment specialist 
positions were eliminated in 2016, and responsibilities for employer partnerships 
shifted to the grant coordinator. Likewise, three different staff members were 
charged with collecting data on participants at various points during the grant 
period. Like the coordinators, the new data administrators required time to 
familiarize themselves with the project. 


Turnover at the college was compounded by turnover at WIN, Southwest’s partner in 
administering the IRT program. According to a current grant staff member, “The 
consistent theme is constant turnover.” He advised that everyone on the grant team 
be familiar with what the grant coordinator is doing and his or her rationale for each 
decision-making process. He noted that strong documentation and communication 
are critical to the success of federally funded grants similar to Solutions. 


Strengthened Institutional Capacity 


Despite the challenges above, developing and implementing the IPCT career pathway 
has strengthened the college’s institutional capacity, largely through the availability 
of new programs and the purchase of new equipment to support those programs. 
Under the grant, Southwest also developed new short-term training to provide 
additional workforce development opportunities for unemployed and 
underemployed students as well as incumbent workers seeking to advance at their 
companies within the process control field. 


New Curricular Offerings 


The Solutions grant established a career pathway for process control technicians, for 
whom employer demand is growing in the greater Memphis area. In the fall of 2014, 
Mid-South Community College in West Memphis, Arkansas (now known as Arkansas 
State University (ASU)-Mid South) and Southwest held a Greater Memphis Process 
Technology Forum, which included an informal survey of the needs of 15 process 
control employers. Annual hiring needs averaged a minimum of 80 open process 
control positions per year, with 12 of the 15 employers expecting demand to increase 
in the future (The Manufacturing Institute & Thomas P. Miller, 2015). The Solutions 
program established Southwest as a training site for process control technicians. 
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The majority of IPCT courses are based on the Amatrol program, an online applied 
technical training system. Students complete web-based modules at their own pace. 
The onsite technologies laboratories at Southwest let students apply the skills from a 
specific module on corresponding Amatrol equipment. Technologies program faculty 
members and laboratory technicians are available to offer additional instruction and 
answer students’ questions. At the end of each instructional unit, students take a 
hands-on assessment using Amatrol equipment. Technology faculty or staff 
members are available to students in the Solutions laboratory from early morning 
until the evening to accommodate students who work a range of job shifts. 


Since its implementation, Southwest faculty members have revised the for-credit 
program to better meet the needs of Southwest students, while ensuring it remained 
consistent with the existing program at NSCC. They reorganized instruction into 
shorter module units within IPCT courses. According to the technologies 
department’s IPCT program coordinator, the new curriculum organization allows 
students to gain specific skills and apply them on the job more quickly than the 
original approach. The ability to apply skills on the job is especially important 
because, according to the coordinator, the majority of Solutions students are already 
working in manufacturing settings. The IPCT coordinator contrasted the student 
population at Southwest with that of NSCC, noting that students at Southwest were 
more likely to be older and employed full time while NSCC was “more of a trade 
school with traditionally-aged college students.” 


The IRT program, the first step in the career pathway, fills an important role in 
upskilling students who are not academically and technically prepared for for-credit 
programs in process control. Many students who enroll at Southwest and similar 
institutions lack basic skills in reading, math, and technical literacy. Through the 
expansion of the IRT, Southwest could serve as a regional training hub for workers 
requiring additional basic skills. ASU-Mid South recently developed a process control 
technology program as part of a TAACCCT Round 4 grant awarded to ASU Mid- 
South, Southwest, and two other higher-education institutions. Given the student 
population’s need for basic skills remediation, external consultants recommended 
that ASU Mid-South use Southwest’s IRT course to upskill its students (The 
Manufacturing Institute & Thomas P. Miller, 2015). Through the TAACCCT Round 3 
grants, Southwest has expanded IRT and solidified the program as a regional training 
resource. 


Other Short-Term Training Opportunities 


In addition to supporting sections of IRT, the grant supported employer partnerships 
to offer onsite, short-term training focused on process control technology. In order 
to offer customizable training, Southwest purchased training curricula: the Society 
for Chemical Manufacturers & Affiliates (SOCMA) Chemical Process Operator 
Training, as well as the Basic & Advanced Mechanical-Electrical-PLC-Process Control 
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Training, developed by Scientific Management Techniques. Southwest purchased the 
SOCMA curriculum at the request of Penn A. Kem, a chemical development and 
manufacturing company based in Memphis. 


The grant has supported the development of customized short-term training 
opportunities at Steel Warehouse/SFI, a steel machining and fabrication company, 
and Chemours, a chemical manufacturer. Southwest worked with Steel Warehouse to 
develop a multi-phased curriculum for training technicians to maintain conveyer 
belts designed to carry steel. According to a company representative, applicants for 
maintenance positions tended to have experience in logistics and distribution, given 
the industry’s presence in the Memphis area. Their lack of experience with steel 
necessitated additional on-the-job training. Steel Warehouse partnered with 
Southwest to develop a customized curriculum based on the existing technical 
curricula available at Southwest. The college provides instructors to Steel Warehouse 
who use company equipment to teach the courses. A Steel Warehouse representative 
described the courses as “wonderful,” noting, “We’ve gotten a lot of participation [in 
the classes]. The majority of the professors that Southwest sends out are very 
interactive. The participants seem like they’re enjoying it very much.” 


In the spring of 2017, Southwest began offering short-term workforce development 
training programs supported through Solutions and the other two Round 3 
TAACCCT grants at Southwest. ISO Internal Auditor Training is a two-day workshop 
on the knowledge and skills necessary to conduct internal quality systems audits. 
Participants receive two certifications from ISO at the conclusion of the workshop. 
Lean Six Sigma Yellow Belt training is also a two-day workshop. It focuses on how to 
reduce waste and improve efficiency within industrial settings. Completers receive a 
Lean Six Sigma Yellow Belt certification. 


During the 2016/17 academic year, Solutions focused on recruiting participants for 
the ISO and Lean Six Sigma trainings. The college was successful recruiting 
participants into these programs, compared to for-credit Solutions programs. In 
total, 93 students participated in the trainings in the spring of 2017. According to a 
grant staff member, the short-term training opportunities allowed for more 
completers of Solutions-funded programs, since at the conclusion of the workshops 
students take an assessment and earn a certification. 


According to grant leadership, a disadvantage of ISO and Lean Six Sigma training is 
the short amount of time students are engaged with the college. The short timeframe 
does not allow college staff to establish strong relationships with students or 
effectively track their post-program outcomes. 


New Equipment 


Delays in purchasing equipment for the Solutions laboratory presented an 
implementation challenge for the grant. The lab was not fully equipped until the 
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summer of 2016. Nonetheless, grant funding eventually provided the equipment 
necessary to offer IPCT courses. The college purchased 10 hands-on learning systems 
and pieces of training equipment to support process control technology. Southwest 
acquired the first piece of equipment for the laboratory in February 2016, three 
pieces of equipment in April 2016, and the remaining equipment in June 2016. The 
training equipment supports instruction in a variety of industrial processes related 
to process control technology. 


Engagement with an External Implementation Consultant 


In the fall of 2016, the coordinators of the three TAACCCT grants at Southwest 
partnered with an external consultant to develop a strategic plan for the remaining 
months of program implementation. The consultant, a former paper executive and 
strategic planning expert, had provided technical assistance to Mount Wachusett 
Community College as part of its TAACCCT program. The two-page plan included 
five goals related to participant recruitment, outcome measures, employer 
engagement, and course schedules, followed by a series of concrete actions, 
responsible individuals, and internal deadlines. The plan led to actions in marketing, 
participant recruitment, and employer engagement. Marketing activities included the 
development of promotional materials and a script on the process control program 
for recruitment presentations. The establishment of the TAACCCT 3 Advisory Board 
was also originally outlined in the plan. 


The consultant also assisted with the implementation of ISO and Lean Six Sigma 
trainings and connected Southwest with contacts at the two organizations. Overall, 
grant staff members found the consultant’s assistance very helpful. According to 
one, “We have been able to successfully have a strong push at the end of the grant, 
and it’s all because of him. He came in and streamlined things, and he helped us 
figure out where we needed to go and what we needed to do. He helped us make sure 
we were able to implement the plan and has connected us with resources.” 


Employer Engagement 


The original Solutions grant proposal emphasized developing and strengthening 
partnerships with employers to ensure that the IRT and IPCT programs were well- 
aligned with industry needs. The proposal indicated that the college would establish 
five workgroups, including an employer partner workgroup. Although Southwest 
recruited members for the workgroup, it did not convene. Instead, Southwest staff 
elected to meet with employer partners on an individual basis. Although the 
workgroup was not established, Solutions grant staff members took steps to engage 
employers, particularly during the final year of grant implementation (Table 3). 
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Table 3. Implementation Timeline for Employer Engagement 


Date Solutions Project Milestone 

Spring 2014 Employer partner workgroup membership 
established 

June 2015, November 2015, and February Solutions grant staff members meet with 

2016 industry partners and representatives of the 


Greater Memphis Alliance for a 
Competitive Workforce (GMACW) 


Spring 2016 Southwest begins participating in the 
Process Technology Talent Sector Council 


August 2016 and November 2016 Process Technology Talent Sector Council 
meetings at Southwest 


Fall 2016 TAACCCT 3 Advisory Board established 


Variation from Original Program Design 


Instead of establishing an employer workgroup specific to Solutions, staff members 
asked for feedback from individual area employers. In total, the grant shared 
information about the program with 17 area companies with a potential need for 
process control employees. The college developed more in-depth partnerships with 
several companies that provided feedback on the for-credit courses and equipment 
and reviewed the curriculum as part of the TBR approval process. During the 
2016/17 academic year, the Solutions team increased its efforts to engage employers 
through participation in the Process Technology Talent Sector Council and the 
establishment of the TAACCCT 3 Advisory Board. 


Challenges to Employer Engagement 


A primary challenge for employer engagement was that employer partnerships were 
not established prior to program implementation. The Solutions grant proposal to 
the U.S. Department of Labor was written as though the employer work group was 
already established, which it had not been. Although grant staff conducted outreach 
to individual employers and solicited their advice on the extent to which the program 
aligned with industry needs, wide-scale employer feedback was not collected until 
Southwest began participating in the regional Process Technology Talent Sector 
Council in the spring of 2016 and established the TAACCCT 3 Advisory Board in the 
fall of 2016. 


A lack of employer responsiveness to outreach requests may have contributed to the 
delayed development of relationships. Grant staff found it difficult to incentivize 
employers to visit the college to provide feedback on the curriculum or laboratory 
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equipment because employers tended to have other obligations. For example, nine 
employers responded “yes” to an invitation to participate in the TAACCCT 3 
Advisory Board that the evaluation team observed in March 2017, yet only three 
employers were able to attend the meeting. 


Because a Southwest-specific employer advisory board was not established until 
2016, the college received some employer feedback late in the grant period that 
might have been helpful earlier. For example, during the March 2017 TAACCCT 3 
Advisory Board meeting, employers informed grant staff members about the test 
used to assess the skills of prospective employees (the Ramsay Assessment). This 
information would have been useful during the curriculum planning stages, when 
Southwest could have ensured that the curriculum was aligned to the assessment. In 
addition, members provided feedback on how best to engage with employers. 
Advisory Board members recommended that college staff members visit employers 
on-site, rather than requesting that they come to the college on a regular basis. 


Program Implementation Delays 


The nearly year-long period required to secure approval from the TBR contributed to 
challenges establishing and strengthening partnerships with industrial process 
control technology employers. Prior to the implementation of the for-credit programs 
in the spring of 2016, some employers were reluctant to partner with the college 
because they did not know whether the skillset that students would develop through 
the program would align with their needs. Employers expressed interest in examples 
of graduates of the program or current students working in the process control field, 
which the college was unable to provide. 


Staff Turnover 


According to one employer partner, staff turnover at Southwest contributed to slow 
responses related to offering customized, on-site training courses. The employer was 
frustrated with the turnover, particularly transitions to new workforce development 
leadership, saying: 


it is like we have to start over all the time...what happens is I don’t 
realize someone’s left and I’m waiting on feedback from them. Then, 
through the grapevine, I learn that they’re gone. There’s not that 
introduction where they transition us to the next person. If there’s 
any way to retain their people better, that would be wonderful. 


Despite these challenges, the employer partner reported being pleased overall with 
the short-term training course at the company. The employer had requested that 
Southwest faculty teach the class based on their in-house equipment and schematics. 
Instructors came to the work site to gain an in-depth understanding of how they 
could customize Southwest’s existing curriculum for the employer. The employer 
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representative said, “They really engaged to understand our business and be able to 
teach the students the specifics of what we needed as a company.” She praised the 
instructional team at Southwest for operating as a responsive, “well-oiled machine,” 
despite leadership turnover. The employer partner said that the course was helping 
to fill skill gaps among the company’s workforce and that supervisors and students 
alike were satisfied with the course. 


Employer Engagement Strengths 


One of the goals of the TAACCCT 3 Action plan, developed in partnership with the 
external consultant in 2016, was to reengage local employers in the industries served 
by the TAACCCT grants. In 2016 and 2017, Solutions scaled-up employer 
engagement through involvement with the Process Technology Talent Sector Council 
and the establishment of the TAACCCT 3 Advisory Board. In addition, employer 
partners indicated that they were satisfied with the for-credit process control 
curriculum and that it was well-aligned with their workforce needs. 


Process Technology Talent Sector Council 


The Greater Memphis Alliance for a Competitive Workforce (GMACW) was formed in 
2014 as a joint initiative of the Greater Memphis Chamber of Commerce and the 
Memphis and Shelby County Regional Economic Development Plan, in partnership 
with the Brookings Institution. The mission of GMACW is to ensure that local 
postsecondary education and workforce development programs are well-aligned with 
employers’ needs. GMACW facilitates sector-specific councils, including the Process 
Technology Talent Sector Council. The council articulates career pathways for the 
field and provides feedback to education programs on current and future workforce 
needs. 


Southwest regularly participates in sector council meetings, which were held on the 
college’s campus in August and November of 2016. During each meeting, Southwest 
made an effort to familiarize employers with the process control programs and 
connect Southwest students to process control employers. In August, the grant team 
offered an overview of the IPCT program, as well as customizable work-based 
training in process control. During the same meeting, faculty members offered a tour 
of the process control laboratory. In November, Southwest held a process control 
career fair in conjunction with the meeting. 


Positive Feedback on the IPCT Curriculum 


Overall, employers interviewed for the evaluation were satisfied with the IPCT 
curriculum, although they would make adjustments based on company needs. An 
employer whom Southwest consulted about the curriculum said that new hires 
without experience are unlikely to have training in process control. Onboarding 
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inexperienced workers requires his company to make a large investment in training. 
The employer believed that IPCT would help to fill positions at the company. During 
the interview, the employer said that the company would be excited to hire someone 
who had been successful in the certificate or AAS degree programs because the 
programs teach students what they need to know to be successful in process control. 
He said that the coursework content related to electrical work was slightly more in- 
depth than was needed at his company, but that the content would likely be useful 
for other process control employers in the region. 


Another employer partner indicated that the IPCT program was “headed in a good 
direction that would meet the majority of our needs.” This industry representative 
had visited the campus both before and after Southwest installed the hands-on 
process control equipment in the Solutions laboratory. During the second visit, he 
“saw much greater emphasis on the process industry” and “could tell that there had 
been significant efforts toward developing programs that are more focused on the 
process industry.” The respondent’s company manufactures different kinds of 
chemicals. He noted that the Southwest program placed more emphasis on steady- 
state processing, the type of process control necessary in a facility that processes 
one type of product, such as an oil refinery or a beverage company. He would like to 
see additional instruction in batch processing, but acknowledged that there is a 
greater regional need for steady-state process control technicians. He was pleased 
that Southwest offered to work with his company to develop courses or curriculum 
modules focusing on batch processing. Of the Solutions faculty and staff he said, 
“They have been inclusive and very responsive to my questions. I think they’ve got it 
[the IPCT program] going in the right direction.” 


At the August 2016 meeting of the Process Technology Talent Sector Council, 
employers indicated that they would like technicians to have additional training in 
analyzing the root cause of problems within a process. In response, the grant 
coordinator from Southwest described how the Amatrol equipment in the Solutions 
laboratory has built-in fault codes to simulate the manufacturing environment. 
Employers at the council also said they would like workers to have stronger basic 
mechanical aptitude and reading comprehension skills. During an interview, another 
employer emphasized the importance of strong written communication skills for 
ensuring safety on the plant floor. These comments underscore the importance of a 
career pathway that includes a basic skills course, such as IRT. 


Plans to Scale-up Internship Model 


In 2015, the college partnered with KTG, a paper manufacturing company based in 
Memphis, to develop an electrical and instrumentation internship for students 
enrolled in AAS degrees in AIIT and Engineering Technologies, two for-credit 
programs within the same department at Southwest as IPCT. Students can also enroll 
in IPCT courses while participating in the internship. Students work 20-25 hours per 
week at KTG for 18 months and receive a stipend. Thus far, four students, two 
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incumbent KTG employees and two other Southwest students, have participated in 
the internship. All four have either been hired permanently or advanced up the job 
ladder. Career services office staff at Southwest expressed a desire to learn from the 
program and incorporate additional employers, including those focused on the 
industrial process control technology sector. 


Program Marketing and Participant 
Recruitment 


Student recruitment could be the most significant challenge for the Solutions 
program. IPCT courses were not offered until the spring 2016 semester. As of July 
2017, there have been only five enrollments in IPCT courses, and no students are 
enrolled in either the IPCT certificate or AAS degree. It was particularly challenging 
to recruit students prior to program approval. Although there was outreach to recruit 
students for IRT and AIIT courses that the college intended to count toward IPCT 
credentials, recruitment efforts accelerated in 2016, when IPCT courses were first 
offered (Table 4). 


Table 4. Implementation Timeline for Recruitment Activities 


Date Solutions Project Milestone 
Spring 2016 Website created for the IPCT program 
Fall 2016 Meeting with marketing consultants to 


discuss marketing strategies to support the 
sustainability of the program 


October 2016 National Manufacturing day event for high 
school students 


November 2016 Process control career fair at Southwest 


The Solutions proposal to the U.S. Department of Labor did not include an in-depth 
plan for marketing the program and student recruitment. Thus, it is difficult to 
determine the extent to which grant activities were consistent with the original 
program design. This section details recruitment-related challenges and recruitment 
efforts implemented in the last year of the grant. 
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Implementation Challenges for Recruitment 


Delayed Implementation Timeline 


During the three annual evaluation site visits, grant staff indicated that the slowed 
implementation timeline was the most significant obstacle to effective recruitment 
activities. Despite not yet being able to offer the IPCT program, grant staff employed 
several strategies to increase prospective students’ interest during the first two years 
of the grant. The principal strategy was to visit IRT classes to introduce the process 
technology control field and encourage students to enroll in courses that would be 
cross-listed in the IPCT and AIIT programs. Despite these efforts, students were 
reluctant to enroll in a program that was not officially launched. During interviews 
early in the grant period, staff members said that it was difficult to generate interest 
in process control among students who were not currently working in the field. IRT 
graduates who sought further training at Southwest decided to enroll in other 
technical programs. 


Lack of Understanding of the Process Control Field 


At the beginning of the grant period, Southwest could have benefited from a strategic 
plan for marketing activities because of prospective students’ lack of familiarity with 
the process control field or the job opportunities available within it. One staff 
member reflected on presenting the program to prospective students: “I have to 
specifically break down the Solutions program and what industrial process control is, 
what it does in a company...I] haven’t come across one person who knows what it is 
already. It’s easier to grasp if they’ve been working in the industry.” 


To address potential participants’ lack of knowledge, both Southwest staff members 
and employer partners suggested additional work-based learning opportunities or 
facility tours for high school students. According to one of the participants in the 
March 2017 TAACCCT 3 Advisory Board meeting, there is a “need to convey at a high 
level that manufacturing is not dirty, dangerous work.” 


From 2014 to 2016, there was some outreach to high schools focused on IPCT, 
described in earlier interim reports. The TAACCT 3 Action plan, developed in the fall 
of 2016, included plans for increasing outreach to high schools. Proposed activities 
included tabling outside of area school cafeterias and athletic events, speaking at 
PTA meetings, and meeting with local guidance counselors. As of the spring of 2017, 
grant staff members had not attended events or held meetings at local high schools. 


Marketing Activities in the Final Grant Year 


During the 2016/17 academic year, the Solutions grant team conducted additional 
outreach and marketing activities. A website for the IPCT program launched in the 
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spring of 2016, when for-credit courses were first offered. Southwest plans to 
include graduate testimonials on the website once more students have completed 
coursework. 


To supplement an IPCT brochure created in 2015, grant staff developed a series of 
talking points on process control, one of which describes the field and program: 


Process control is an engineering discipline that deals with 
architecture, mechanisms, and algorithms for maintaining the output 
of a specific process within a desired range. For instance, the 
temperature of a chemical reactor may be controlled to maintain a 
consistent product output. 


The talking points also include information about salary ranges for technicians in 
process control. They conclude with a detailed description of the IPCT technical 
certificate and AAS degree, including the format of the Amatrol coursework and how 
certificate coursework articulates to the AAS degree. In addition to the talking points, 
the coordinators of the TAACCCT grants at Southwest jointly created a slide show to 
promote the for-credit technical programs, including Solutions, entitled “Are You 
Southwest Skilled?” 


In the fall of 2016, Southwest began working with the Redwing Group, a strategic 
marketing firm in Memphis, which has helped the workforce development 
department strengthen its print- and media-based marketing. The grant coordinator 
and the Associate Vice President of Workforce Development conducted radio and 
television interviews in the spring of 2017. Grant staff members are currently in the 
process of developing additional content for the website and content for Memphis 
Works, a website created by GMACW where local residents can learn about regional 
job opportunities and the training required for them. 


Throughout the 2016/17 academic year, the workforce development and academic 
staff members at Southwest held outreach events on campus to promote the 
TAACCCT grants. In October 2016, Southwest hosted a National Manufacturing Day 
event, during which 50 high school students toured the campus’ advanced 
manufacturing facilities, including the Solutions laboratory. In November 2016, 
Southwest hosted a process control technology career fair in conjunction with the 
Process Technology Talent Sector Council meeting held at Southwest. During the fair, 
more than 100 high school and Southwest students had the opportunity to tour the 
laboratory facilities and speak with representatives of 12 process control employers. 
Examples of participating companies included: Coca-Cola, Hershey Foods, Lucite 
International, Memphis GP Cellulose, GlaxoSmithKline, Sonoco, Sparco.com, and 
Valero. 
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Prior Learning Assessments 


The 2013 Solutions proposal outlined a plan for a grant to support the 
implementation of the Learning Counts, an online portfolio assessment system 
developed by CAEL. After an initial meeting with CAEL, grant staff determined that 
implementation of Learning Counts was not feasible. Instead, Solutions engaged to 
develop a PLA guide for staff designed to streamline the process for awarding credit 
for prior learning. The guide includes instructions for awarding prior learning credit. 
It also serves as a central location for the forms staff and faculty members need to 
submit to the college to award prior learning credit. 


In addition to the guide, CAEL worked with Southwest to develop course challenge 
assessments for IPCT coursework. Course challenge assessments are local tests 
designed to verify learning achievement related to a particular course. The full 
implementation of the course challenge exams is ongoing. 


Data and Tracking Participants 


The designers of the Solutions program originally intended to create a 
comprehensive data tracking system to track participants as they moved through the 
process control career pathway. The online dashboard would have allowed students 
to track their progress through the Learning Counts system. Faculty and staff were to 
track students through Amatrol, KeyTrain, and Salesforce. Although the Amatrol and 
KeyTrain systems are available to IPCT and IRT students and faculty respectively, the 
Salesforce system was not purchased because it did not meet the TBR’s security 
requirement. Thus, the timeline for full adoption of a data tracking system was 
delayed (Table 5). In addition, the definition of a grant participant was broad, 
creating confusion about which students intended to enroll in the IPCT program. 


Despite data challenges, grant staff took steps to ensure the sustainability of future 
collection of student data by transitioning the management of student files and 
services to the student and career services offices at the college. 


Table 5. Implementation Timeline for Activities Related to Data and Tracking Students 


Date Solutions Project Milestone 

Spring 2014 Initiation of procurement process for 
Salesforce 

Fall 2014 TBR indicated that it must approve the data 


tracking tools purchased 
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Date Solutions Project Milestone 

Spring 2015 TBR determined that Salesforce cannot be 
purchased. Grant staff considered other 
options. 

Summer 2015 Selection of the G*Stars system for tracking 


grant participants 
Fall 2015 Implementation of G*Stars 


November 2016 Workforce development staff transitioned 
participant case-management 
responsibilities to the student services and 
career services offices. 


Prior to the submission of the proposal to the Department of Labor, the Solutions 
grant team would have benefited from an initial planning period to assess the 
feasibility of purchasing the Salesforce system and using it for data tracking. Had 
appropriate background research been conducted, the design team might have 
anticipated that the college would need to seek approval from the TBR to purchase 
Salesforce and that it might not meet the organization’s security requirements. 
Because the college was not aware of these requirements, the implementation of a 
TAACCCT-specific data system was delayed by more than a year. 


Definition of a Participant 


The U.S. Department of Labor’s definition of a participant in a TAACCCT-funded 
program is broad. Participants can include students who enroll in a course supported 
with grant funds. The college intended for some of the courses in the IPCT technical 
certificate and AAS degree to overlap with requirements for AIIT, another program 
within the technologies department. Prior to the official launch of the IPCT program, 
Southwest counted students who enrolled in these overlapping courses as 
participants, asking them to complete a grant intake form. However, the intake form 
did not include questions about whether the student intended to pursue a credential 
or employment in the process control field. Thus, it was impossible to determine the 
level of treatment for each participant. It is possible that all students counted as 
participants were pursuing an established certificate or AAS degree in AJIT rather 
than the IPCT program, which was the focus of the grant. 


Transition of Data Responsibilities to Other Offices 


In November 2016, the workforce development staff transitioned tracking of student 
data and case management to the student services and career services offices. The 
purpose of the transition was to support the long-term sustainability of the grant’s 
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data system and to ensure students’ continued access to services. The student 
services office is responsible for advising and academic support, while career 
services follows up with students to track their employment and offers career 
planning support such as assistance with mock interviews and resume development. 


Solutions grant staff identified the for-credit students who were still enrolled and 
forwarded their files to student services. The student services office is working to 
expand intrusive advising to all students. Intrusive advising is a process of ongoing, 
proactive contact intended to make sure students are on track academically and 
motivated to persist in college. The student services office reaches out to students 
via e-mail, phone, and text to ensure that they are academically on track. As of March 
2017, the office had conducted two rounds of outreach to Solutions students. 


Similarly, career services staff have conducted multiple rounds of outreach to 
students who have completed the IRT program or are no longer active in for-credit 
coursework. A major goal of this outreach is to track employment outcomes for the 
TAACCCT grants. Staff members have reached out to all TAACCCT participants to 
see where they are employed and whether they secured new employment or received 
a pay increase after completing a program or leaving Southwest. A second purpose of 
the outreach is to invite Solutions participants to take advantage of the college’s 
career services, such as interview preparation, resume assistance, and ongoing career 
fairs. Several students expressed interest in returning to the college to continue their 
educations. The career services office refers these students back to student services. 


The career services office sends information on employment outcomes back to the 
workforce development staff for grant reporting purposes. The grant has 
documented employment outcomes for 23 current and former participants, including 
19 IRT completers and four AIIT students. Of those, 13 (54 percent) reported being 
employed at Smith & Nephew. Several students shared stories of how they gained 
employment or received a wage increase after completing IRT. For example, one 
student wrote after a job interview, “[The company] said I did excellent and that they 
wanted to offer me a position as a line technician with a starting pay rate of $16.19 
an hour with shift differential.” 
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Outcomes Study: Descriptive Results 


Due to the delayed implementation of Southwest’s Solutions program, we are unable 
to run the initially planned analyses. As there have been only five IPCT course 
enrollments, some of which may be the same students taking different courses, there 
is no reliable way to estimate the Solution program’s impact. Systematic data on 
these students have not yet been collected or made available to CNA, making even a 
description, let alone any attempt at a causal impact of the impact of IPCT 
enrollment, impossible. 


Even with the available data, we cannot perform a rigorous impact analysis. Part of 
this is for methodological reasons. Without valid treatment and comparison groups 
for the for-credit IPCT students, we cannot run regression-based analyses that will 
lead to causal conclusions. Available data also limit the questions that we can 
answer: too few students report data on employment or wages to reliably estimate 
the impact of AIIT or IRT courses on these outcomes (especially since whether a 
student reports his or her wage or employment status may depend on the wage or 
employment status themselves). 


As a result, we instead provide a high-level descriptive overview of participant 
characteristics and of some medium-term outcomes for IRT enrollees and AIIT 
enrollees. The results are drawn from program enrollment and participation data 
provided by Southwest. 


Demographic and Enrollment Statistics 


In total, there were 407 Solutions grant participants. Of these, 176 participated in the 
non-credit IRT program designed to prepare students for entry-level manufacturing 
jobs, while 107 were enrolled in the for-credit Advanced Integrated Industrial 
Technology (AIIT) program.’ An additional 93 students were enrolled in ISO or Lean 
Six Sigma certification courses, and 31 students took other courses or programs at 


> Our data came from multiple data sets, and several students were listed as AIIT participants 
in one data set but as participants in other courses in another data set. We have counted these 
students as AIIT participants for the purposes of this report. 
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the college. We present separate summary statistics for all students who completed 
intake forms, IRT students, AIIT students, ISO and Lean Six Sigma students, and 
other students (Table 6). The first set of columns includes all students across all 
listed programs. The second set of columns contains only IRT students, and the third 
contains only AIIT students. The fourth contains both ISO and Lean Six Sigma 
students, and the fifth contains all students not captured in any of the categories 
above. 


The average student completing an intake form was approximately 37 years old. 
Nearly two-thirds of these students are male, and nearly three-quarters of 
respondents (those students providing information for a particular survey item) are 
African American. Eight percent of respondents are veterans, while less than one 
percent are spouses of veterans. Approximately two percent of respondents 
indicated some form of disability, though disability type is not observed in the data. 
Nearly half of respondents were employed; however, it is likely that this 
overestimates the employment rate among all students who filled out intake forms 
because data on employment were missing for 56 respondents. No students in our 
sample were eligible for TAA benefits. 


Table 6. Demographic and Enrollment Characteristics of Solutions Participants 


All Students IRT AllT ISO/Lean Six | other 
Sigma 

Characteristic N N N N N 
sete DORA 37.4 | 4061349 | 176 | 357 | 107/432 |93 | 406 | 30 
Percent Male 65.8% | 406 | 60.8% | 176 | 95.3% | 107 | 31.5% | 92 | 93.5% | 31 
Percent African | 747% | 376 | 91.9% | 173 | 49.4% | 83 | 79.8% |89 | 32.3% | 31 
American 
Percent Veteran 8.0% 339 | 6.6% 167 | 10.5% | 57 7.9% 89 11.5% | 26 


Percent Veteran | 93% | 310 | 0.0% | 143 | 1.9% |53 |00% |8s8 |o0% | 26 


Spouse 
Percent Disabled | 1.8% | 339 | 1.2% |171 118% |57 134% 189 100% | 22 
Seieeds 49.6 | 351 | 35.3% | 173 | 328% | 58 | 71.9% | 89 | 96.8% | 31 
Employed 
Percent Enrolled | 41¢ | 386) NvA | N/A 115.0% | 107 | N/A | N/A | 0.0% | 31 
Full-Time 


Students enrolled in IRT courses are two-and-a-half years younger than students 
overall. They are approximately 5 percentage points less likely to be male and 17 
percentage points more likely to be African American than all Solutions participants. 
They are approximately 14 percentage points less likely than the overall sample to be 
employed. 


AIIT enrollees are about a year older on average than IRT enrollees. They are 
overwhelmingly male—fewer than five percent are female. Approximately half are 
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African American. AIIT enrollees are only two-thirds as likely as students overall to 
be employed. 


ISO and Lean Six Sigma enrollees are over eight years older than IRT students on 
average. Under a third are male, while nearly 80 percent are African American. Over 
70 percent of short-term certification seekers are employed. 


A small group of students were listed as Solutions participants despite not taking the 
IRT, AIIT, ISO, or Lean Six Sigma programs. Of these students, 23 took courses in 
Industrial Electrical Training, while 7 took courses for Welding Certification. These 
students were over 5 years older than IRT students on average. Over 90 percent were 
male, and less than a third were African American. Nearly all of these students are 
employed. 


IRT WorkKeys Results and Program 
Completion Rates 


Although we cannot analyze the impact of the IRT program on employment or 
wages, we were able to combine program data for IRT students with data on pre- and 
post-course WorkKeys assessments. IRT students took three WorkKeys assessments 
at both the beginning and the end of their program: Applied Mathematics, Locating 
Information, and Reading for Information. Their post-scores determined whether and 
what level of National Career Readiness Certificate (NCRC) they earned. The Applied 
Mathematics assessment focused on the problem solving and mathematical 
reasoning skills necessary for math-related situations that occur in the workplace. 
The Locating Information assessment measures participants’ ability to locate, 
synthesize, and use information in graphic documents found in the workplace. 
Finally, the Reading for Information assessment focuses on participants’ ability to 
identify and use information in written documents, such as memos, directions, and 
workplace policies. 


On each of the three assessments, the largest group of students experienced no 
change in their score levels. However, a sizeable group of students increased their 
scores on each assessment—33.6 percent in Applied Mathematics, 16.8 percent in 
Locating Information, and 28.4 percent in Reading for Information. As Locating 
Information was the subject with the fewest score increases and the most score 
decreases, IRT instructors might consider investing additional time on skills related 
to Locating Information or reassessing the effectiveness of their instructional 
approach. 
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Detailed results are provided in tables 7 through 12. Separate sets of tables present 
results for scores on the Applied Math (tables 7 and 8), Locating Information (tables 
9 and 10), and Reading for Information (tables 11 and 12) assessments. 


In Table 7, students are sorted on a grid of pre- and post-test score level in Applied 
Mathematics. Each row represents a different pre-test score level, while each column 
represents a different post-test score level. Cells in the table are shaded based on the 
degree of improvement or decline participants experienced. Greater degrees of 
improvement are shaded in brighter green, while greater degrees of decline are 
shaded in darker red. Cells showing no change between pre- and post-scores are 
white, while cells corresponding to missing pre- or post-test values (which cannot be 
compared) are shaded in gray. 


In Table 8, this information is condensed into a two-row table showing the percent of 
students whose score levels decreased by a particular number of levels, shaded to 
correspond to Table 7. The report presents both tables because Table 8 is more 
compact and easier to interpret, while Table 7 shows where particular gains and 
losses occur, which may be of value when prioritizing student supports. Values in 
Table 8 will not add up to 100 percent, since approximately 12.3 percent of students 
had missing values on either the pre-test or the post-test. 


Table 8 shows that approximately 43 percent of students had the same score level on 
both the pre- and post-Applied Mathematics assessments. Just under 30 percent 
increased by one level, while 9 percent decreased by one level. Approximately 4 
percent increased by two levels, and 2.6 percent decreased by two levels. No students 
improved or declined by more than two levels. 


Table 7. Level Changes in WorkKeys Applied Math Pre-Test/Post-Test Scores 


Post-Test Score 

<3/3 | 4 7 | Missing | Total 

<3 o|3 i 1 4 

3 3] 10/9 3 a7 

4 7 24 4 6 7] 

Pre-Test | 5 my 4 mC 4] 
Score | 6 Momo 4 |2 1 7 
7 —sEO 0 1 

Missing | 1 ] ] i_|_© |_@ fe) 4 

Total | 7 | 21 | 38/59] 13] 2 15 155 
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Table 8. Summary of Level Changes in WorkKeys Applied Math Pre-Test/Post-Test 
Scores 


Difference | Decrease | Increase 
O Levels 42.6% 

1 Level 9 29.7% 
2 Levels Pe 
3+ Levels 


Table 9 and Table 10 show corresponding score level patterns on the Locating 
Information assessment. On this assessment, approximately 57 percent of students 
kept the same score level, with nearly half of all students scoring at level 4 on both 
exams. No IRT students scored at levels 6 or 7 on this section. Approximately 14 
percent fell by one level, while 16 percent rose by one level. 


Table 9. Level Changes in WorkKeys Locating Information Pre-Test/Post-Test Scores 


Post-Test Score 


<3] 3 | 4 | 5 | Missing | Total 
S 0 7 
1 24 

Pre-Test 7 12 107 
Score ee | 4 2 13 
Missing | 1 @ |S || © 0 4 

Total 2 | 30] 97} 11 15 155 


Table 10. Summary of Level Changes in WorkKeys Locating Information Pre-Test/Post- 
Test Scores 


Difference | Decrease | Increase 
O Levels 
1 Level 
2 Levels 
3+ Levels 


Table 11 and Table 12 show how IRT students performed on the Reading for 
Information section of WorkKeys. Approximately 45 percent of all students kept the 
same score level, just over a quarter increased by one level, and approximately 13 
percent decreased by one level. Among students who improved or declined, the most 
common movement was from level 4 to level 5, followed by from level 5 to level 6. As 
before, no students’ scores rose or fell by more than two levels. 


34 


ANALYSIS & SOLUTIONS 


Table 11. Level Changes in WorkKeys Reading for Information Pre-Test/Post-Test Scores 


Post-Test Score 
Missing | Total 
0 3 
On o 98 
i= 69 
ia 7 [2 i 20 
of 0) 1 
Missing | 0 | 2 | 1 ie) fe) 4 
Total 3 | 43 | 65 | 25 | 4 15 155 
Table 12. Summary of Level Changes in WorkKeys Reading for Information Pre- 


Test/Post-Test Scores 


Difference | Decrease | Increase 
O Levels 44.5% 

1 Level 13.4% 25.8% 

2 Levels 1.3% | 2.6% 


3+ Levels 


Table 13 and Table 14 show which levels of the National Career Readiness Certificate 
(NCRC) students earned, based on overall WorkKeys scores. Students whose scores 
were listed as “missing” or as “NC” are shaded in gray on these tables, as these 
scores cannot be compared; nearly a quarter of students had a missing score on both 
the pre- and post-assessments. Over 50 percent of students earned the same 
certificate level, with nearly 40 students achieving a silver NCRC based on both pre- 
and post-assessments. Approximately 13 percent of students increased their 
performance by one level, while 3 percent fell by one level. 


Table 13. Level Changes in National Career Readiness Pre-Test/Post-Test Scores 


Post CRC 
NC Bronze Silver Gold (missing) | Total 
NC ] 8 0) 0) 0) 9 
Bronze 0) 14 15 feaiee| 0) 29 
Silver ] 3 61 5 2 72 
PISTCRS | Gold 0 = 0 i 
(missing) | 1 ] ) 0) 36 38 
Total 3 26 77 11 38 155 
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Table 14. Summary of Level Changes in National Career Readiness Pre-Test/Post-Test 


Scores 
Difference | Decrease | Increase 
0 Levels 52.3% 
1 Level 2.6% | 12.9% 
2+Levels | 00% | 0.0% 


IRT Certificate and Program Completion Rates 


Of the 155 students with WorkKeys scores, 114 earned a NCRC. In total, 7 percent 
(N=11) earned a Gold NCRC, fifty percent (N=77) earned a Silver NCRC, and 17 
percent (N=26) earned a Bronze NCRC. 


Of the 155 students with WorkKeys scores, 120 (77.4%) also had IRT completion 
certificate or OSHA certification completion data. In total, 114 (73.5%) received both 
an IRT and ten-hour OSHA certification, two (1.3%) received just an IRT certificate, 
two received just ten-hour OSHA certification, and two received neither. Thirteen 
students had IRT certificate data but no WorkKeys data.’ 


Of 176 students with IRT intake data, 123 (69.9%) were listed in the data as 
completing an IRT certificate, an OSHA certificate, or both. Of those that were not 
listed as such, seven (4.0%) appeared in the completion data but had no date of 
completion or certification type listed, while 46 (26.1%) did not appear in the 
completion data at all. Of the 123 students that received some form of certification, 
117 (66.4%) received an IRT certificate, 119 (68.8%) received a ten-hour OSHA 
certificate, and 115 (65.3%) received both certificates. 


AllT Credits and Persistence 


Although we do not have assessment scores for AIIT students, we are able to 
examine credit patterns over the two school years for which we have data. These 
data cover parts of the 2014 and 2015 school years; we do not have any data on 
credits earned from fall 2015 onwards. These patterns also let us make some 
inferences as to whether students are persisting from one year to the next. 


* Of these students, two had both IRT and ten-hour OSHA certification, four had just ten-hour 
OSHA certification, and seven had neither. 


36 


ANALYSIS & SOLUTIONS 


Of the 58 students for whose intake data we were able to merge with credit data, 48 
(82.8%) earned credits in either year. Separating enrollment by year, 24 (41.4%) earned 
any credits in year one (covering the spring and summer terms of 2014), while 41 
(70.7%) earned any credits in year two (covering fall 2014, spring 2015, and summer 
2015). Five students attempted a positive number of credit hours in year one but did 
not earn any credits, while seven students attempted but did not earn credits in year 
two. There were 17 students (29.3%) who earned credits in both years, leading to a 
for-credit persistence rate of 17/24, or 70.8 percent. Although the number of AIIT 
students included in this study is small, the retention rate is more than 18 
percentage points higher than Southwest’s overall freshman to sophomore retention 
rate of 52.5 percent in 2015/16 (Tennessee Higher Education Commission, 2016). 


Readers should be careful to avoid conflating the for-credit persistence rate with the 
overall persistence rate, as our data do not allow us to observe why students do not 
earn any credits in a given year. For example, if we assume that students with zero 
credits in a given year were not enrolled, this would mean that the for-credit and 
overall persistence rates are equal; however, if we assume that students with zero 
credits were all enrolled in non-credit courses, we would have an overall persistence 
rate of 100 percent. Thus, although we cannot presently form any conclusions about 
overall persistence, we can at least determine whether students who earned credits in 
year one went on to earn credits in year two.° 


The average number of credits earned (among the 48 students who took any credits) 
is 10.3. We restrict our analysis to students who earned credits in a given time 
period, as the decision of some students to enroll in a non-credit program should not 
factor into any analysis of how intensely for-credit students pursue their studies. 
The 24 students who enrolled in year one earned an average of 13.5 credits, while the 
41 students who enrolled in year two earned an average of 11.6 credits. However, 
students who enrolled in both years earned 17.8 credits, 7.7 more than the combined 
number earned by students who took classes in year one only and year two only. 
This implies that students who earned credits in both years earned more credits in 
each year than students who took credits in that year alone. 


> Complications may arise if some students do not persist because they complete the AIIT 
program. 
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Conclusion: Lessons Learned from 
the Implementation of Solutions 


The core component of the Solutions grant, the stacked and latticed credential 
program leading to a technical certificate and AAS degree in industrial process 
control, was developed and offered to Southwest students as a result of the Solutions 
grant, yet Southwest experienced challenges throughout the grant period that 
impeded the full implementation of proposed grant activities. 


The grant proposal could have benefited from additional preparatory work prior to 
its submission to the U.S. Department of Labor. According to current grant staff 
members, additional consideration should have been given to whether the program 
was designed in a way that aligned with the needs of prospective students and 
employers. In addition, the grant would have benefited from a program 
implementation planning period prior to the start of the grant period. Such a 
planning period would have allowed for the development of an in-depth 
implementation timeline and contingency plans for possible implementation 
challenges. Other activities that would have benefited the grant include earlier 
engagement with employer partners and the development of clear lines of authority 
related to project administration and succession plans. This section briefly discusses 
each of these lessons and how additional activities might have supported 
implementation. It concludes with a discussion of ongoing sustainability planning 
efforts and an overall summary of grant implementation. 


Considerations for Program Design 


Southwest experienced a challenge with recruiting students to participate in the for- 
credit process control technology programs. Although the delayed implementation 
timeline made recruitment especially difficult, Southwest did not consider whether 
the program was aligned with the level of commitment that students were likely to 
make to for-credit certificate and degree programs. According to grant staff 
members interviewed during the 2017 site visits, the design of the Solutions program 
did not take into account the barriers that students face. In addition to being 
unfamiliar with the process control technology field, prospective students may be 
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transient. Many older students must prioritize work and family obligations and may 
not have one, two, or more years to commit to earning a credential. 


Current grant staff suggested that the Solutions program should have included a 
non-credit process control program, similar to the short-term training programs that 
Southwest developed for employers. The non-credit program could articulate into a 
for-credit certificate program. According to one of the respondents, “we need to let 
students exit with something, then be able to come back for a credential.” 
Respondents believe that such an approach would have been beneficial for both 
students and employers. Short-term, non-credit programs can help employers fill 
short-term vacancies. Similarly, such programs help students secure employment 
quickly while still providing them with the opportunity to pursue a for-credit 
certificate or AAS degree. 


Pre-Implementation Program Planning 


The Solutions grant would have benefited from an in-depth program planning period 
prior to the submission of the grant proposal to the U.S. Department of Labor. A 
planning process would have allowed program designers to develop a realistic 
program timeline, assess whether the college had the resources to implement all 
aspects of the program effectively, and plan for the possible simultaneous 
implementation of three Round 3 TAACCCT grants. 


Through the planning process, staff members could have anticipated challenges and 
developed contingency plans. Areas of focus could have included programmatic 
decisions related to data systems and curricula as well as development of a strategy 
for recruitment and marketing. In-depth pre-planning may have mitigated some of 
the implementation challenges that the grant ultimately faced. 


During the pre-program planning period, program designers could have made 
concrete decisions regarding the development of a curriculum for the for-credit 
programs. In reality, the draft curriculum was not submitted to the TBR until the 
spring of 2015, a year and a half into the grant period. It took nearly an additional 
year for the program to be approved. If the draft curriculum had been completed and 
submitted to the TBR at the award of the grant, classes likely would have been 
offered earlier. 


During the pre-program planning period, Southwest could have developed a plan to 
recruit students into the for-credit IPCT programs as soon as they were offered. The 
Solutions proposal did not include a recruitment plan and relied heavily on the 
assumption that IRT completers would transition into the for-credit programs, which 
they did not. In general, grant staff found that students who did not already work in 
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process control were unfamiliar with the field or career opportunities within it. This 
challenge highlights the need for a detailed marketing and recruitment plan. 


Finally, pre-planning could have allowed program designers to anticipate roadblocks 
and develop contingencies. For example, planners could have consulted 
representatives from the TBR to learn about the board’s security requirements for 
data systems and planned to purchase one that met those requirements. Such 
planning would have allowed Solutions to have a consistent data tracking system in 
place several months prior to when G*Stars was ultimately purchased. The pre- 
program planning period would have also allowed for the development of a 
contingency plan for a protracted TBR approval process. 


Earlier Engagement with Employers 


The Solutions proposal called for the development of an employer workgroup, yet an 
employer advisory board for the TAACCCT grants at Southwest was not established 
until 2016. Although Southwest did engage employers to review the Solutions 
curriculum and employers were generally satisfied with it, sustained employer 
engagement could have been a focus much earlier in the grant. This would have 
allowed Southwest to adjust courses based on employer needs. 


The establishment of an employer advisory council and an initial meeting could have 
been part of the pre-program planning process, detailed above. At that time, 
Southwest and participating employers could have discussed the level of 
commitment required of the advisory council and the best approach for gathering 
input from members. 


Establishing an advisory council earlier would have allowed for ongoing, in-depth 
collection of employer feedback on the extent to which the IPCT program aligned 
with labor market needs. Process control employers could have provided regular 
updates on their labor market forecast, including the number of current openings 
and the number of vacancies that they anticipated in the coming months and years. 
A well-established advisory council could have also regularly shared information with 
Solutions staff about the skills they expect from prospective employees. For example, 
at the TAACCCT 3 Advisory Board meeting in March 2017, employers shared 
information about the Ramsay assessment they require applicants to take. The 
TAACCCT programs at Southwest could have benefited from such information 
earlier, perhaps integrating preparation for the assessment into coursework. 


A longstanding employer advisory council may have also supported recruitment 
efforts. Employers might have been able to recommend the program to incumbent 
workers who were interested in building their skills. They also potentially could have 
shared information about the program with their professional networks. 
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Establishment of Lines of Grant Administrative 
Authority and Succession Plans 


During site visits in all three years, confusion about who was in charge of decision- 
making emerged as an implementation theme. The grant was administered by the 
non-credit workforce development department but housed in the for-credit 
technologies department. The grant coordinator with the longest tenure did not feel 
empowered to make decisions without consulting others. Grant staff should have 
developed an organizational chart with clear lines of authority and communication 
across departments. 


Given that slow responses from departments that did directly administer the grant 
were sometimes an issue, the organizational chart could have also specified whom to 
reach out to if a response was not received promptly. The organizational chart 
would benefit both people listed on the chart, by helping them understand their 
responsibilities, and people trying to engage with grant decision-makers at 
Southwest. 


Perhaps most importantly, the grant coordinator and the director of workforce 
development should have been established as grant leaders and been given the 
ultimate responsibility for all grant-related decisions. It would have been beneficial 
for the grant coordinator to have direct contact with the project officer at the U.S. 
Department of Labor rather than establishing the procurement specialist at the sole 
point of contact. Then the grant coordinator could have followed up more frequently 
with the project officer on the status of equipment purchase requests and other 
grant administration matters. If the grant coordinator role had been clearly 
established as a leadership role, perhaps the person fulfilling the position would 
have felt more empowered. 


Given the large amount of staff and leadership turnover that Solutions experienced, 
the grant would have benefited from the development of succession and continuity 
plans. The plans could have detailed how to onboard the next person and ensure that 
ongoing conversations, such as those with employers, continued. Each plan could 
have outlined the staff member’s roles and responsibilities. 


Sustainability Plan Development 


Throughout the March 2017 site visit, respondents emphasized the need to develop a 
strong sustainability plan. Although the grant increased the capacity of Southwest 
through new curricular resources and equipment, staff sought to ensure that the for- 
credit programs thrived after the conclusion of the grant. One respondent said, “How 
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do we ensure that we have a workforce that is trained as the result of the 
investment? How do we create an employer pipeline for process control?” 


As of June 2017, Solutions grant staff members were in the process of developing a 
sustainability plan using the TAACCCT Sustainability Toolkit from the TAACCCT 
Learning Network website. The goal of the sustainability plan is to ensure the 
ongoing implementation of the process control stacked and latticed credentials. 


There have been initial challenges with sustainability. For example, there have been 
no new Solutions IRT sections since the end of grant programming in the spring of 
2017. Grant staff members emphasized the importance of sustained efforts to 
market the non-credit and credit programs to employers and students. 


Implementation Summary 


Overall, the Solutions TAACCCT grant increased the capacity of Southwest Tennessee 
Community College to offer sub-baccalaureate credentials in the in-demand field of 
industrial process control technology. The stacked and latticed credential program 
proposed in the original grant proposal was developed and offered to students as a 
result of TAACCCT funding. The grant funded the scale-up of the IRT program, 
which provides students with the academic, technical, and workforce readiness skills 
necessary to succeed in entry-level employment or transition to a for-credit 
technologies program at Southwest. As a result of grant funding, Southwest invested 
in the equipment and instructional resources necessary to offer a for-credit 
certificate and AAS degree in IPCT. In the fall of 2017, the college will offer several 
IPCT courses, including Introduction to Process Control Technology, Millwright I, 
Quality, and Process Control Technology I. Employer partners have offered positive 
feedback on the program, indicating that content was well-aligned with the skills 
they were seeking in new hires. 


Several challenges precluded the full implementation of the grant activities and 
program components that Southwest outlined in the original Solutions proposal. The 
most significant challenge was the delayed implementation timeline. The college 
intended to implement the AAS degree in the fall of 2014, but TBR did not approve 
the program until late 2015, and IPCT courses were not officially offered until the 
spring of 2016. The delayed approval presented challenges for the recruitment of 
students and employer partners. In addition, the college did not incorporate ISA and 
Kauffman Foundation certificates into the credentials because attention was focused 
on launching the for-credit programs. 


Administrative regulations prevented the implementation of other program 
components. For example, the data tracking system that Southwest proposed did not 
meet TBR’s security requirements. Thus, implementation of a data system for the 
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grant was delayed. A final challenge to implementation was how the grant was 
administered. For much of the implementation period, the grant coordinator had no 
direct contact with the project officer at the U.S. Department of Labor. There was 
also confusion about whether the workforce development department or the 
technologies department was ultimately responsible for grant-related decisions. 
These administrative challenges likely slowed the progress of implementation. 


In the final year of the grant, staff members made a concerted effort to strengthen 
implementation. Southwest engaged an external strategic planning consultant with 
expertise in the manufacturing field. He worked with all three TAACCCT grant 
coordinators to develop a plan for strengthening recruitment and employer 
engagement efforts. Staff members partnered with the Redwing Group to develop 
new marketing materials and increased outreach to prospective students through a 
series of in-person career fairs and information sessions. Southwest also significantly 
increased employer engagement through the establishment and regular meetings of 
the TAACCCT 3 Advisory Board. The transition of data collection and case 
management to the student services and career services office helped ensure the 
sustainability of grant activities. 
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Appendix: Example of the Interview 
Protocol 


Interview Protocol for Southwest Solutions Counselor 


Start with introductions and a brief description of the study. 


Thank you for agreeing to discuss your experience as the Solutions counselor. CNA is the third- 
party evaluator for Southwest Solutions. 


Before we begin the interview, | want go over consent procedures and have you sign this consent 
form if you agree to participate. The information you provide will be used to draft the final report 
on the Solutions program that will describe the views of Southwest Solutions staff, faculty, and 
students. It will suggest ways to make Southwest Solutions more effective and highlight strong 
practices. 


Your comments will be kept strictly confidential; your name or any other identifying 
characteristics will not be associated with specific comments. The interview will take about 20 
minutes. You can stop the interview at any time or choose not to answer specific questions. 


Do you have any questions about the study, interview, or consent form? 


Is it OK if | record our conversation? I'll only use the recording to ensure CNA captures what you 
said accurately. 


Role 


1. Describe your current role within Southwest Solutions? 
Probes: 
e What's a typical day like for you? 


2. Have your professional responsibilities changed over the course of the grant? 


Probe: If so, tell me more about that. What were the biggest challenges to taking on 
new roles? 


Recruitment 


3. Tell me about how students are currently recruited. 
Probe: 
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e How does Southwest partner with other organizations (e.g., WIN) during 
the recruitment process? 
e How does Southwest partner with employers? 


4. What have been the most significant changes to the recruitment strategies during 
the grant period? 


5. What have been the most successful recruitment strategies? 


Students Served 
6. How do you help a student determine which program or credential he or she should 
pursue? 
Probes: 
e Do students take a skill assessment prior to enrolling in the Solutions 
program? 


e If so, how are the results used? 


Support and Counseling Services 


7. How do you work with students once they have enrolled in the program? 
Probes: 
e What guidance or information do you share with students? 
e Which counseling strategies do you use? 


8. What challenges or barriers to success do students face when they enter the 
program? During the program? 


9. How do you work with students to address the challenges they face? 


10. Tell me about the career guidance and placement services students receive from 
Southwest. 
Probes: 
e What kind of career guidance do current students receive? How is it 
tailored to their needs? 
e Arethere changes you would suggest for the program’s approach to career 
services/guidance? 


11. What academic and social support services are available to students (e.g., tutors, 
supportive clubs, etc.)? 
Probes: 
e To what extent do students use these resources? 
e How helpful do students perceive these services to be? 


12. What reasons do students who have dropped out of or stopped the program cite for 
leaving? 
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Relationship with program faculty and staff 


13. How do you collaborate with grant staff? 


14. How do you collaborate with faculty? 
Probes for both questions: 


What information do you share? 

How do you support each other’s work? 

How do you work together to address students’ needs? 
Have there been any issues or challenges with collaboration? 


Program Implementation 


15. Do students have a clear understanding of what they are expected to be able to do 
once they complete their program? 


16. How do employers contribute to program development and implementation? 


17. How do other partner organizations contribute to program development and 
implementation? 
Outcomes 


18. Which aspects of the Solutions program have been most beneficial to students, and 
why? 


19. Which aspects of the Solutions program have been /east beneficial to students, and 
why? 


20. Of those students you worked with who completed the IRT, what factors 
contributed to their success? 
Probes: 
e What, if any, barriers to program completion did these students face? 
e How did they overcome them? 


21. What information do you have about the job performance of Solutions graduates, 
and how Solutions might have contributed to that performance/success? 


Probe: Have there been any changes to the grant program as a result of this 
feedback? 


22. Are there IRT completers now enrolled or planning to enroll in the IPCT? 
Probe: 
e What else could Solutions do to encourage students to continue their 
education? 
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Overall Impressions 


23. What additional resources or information, if any, would help you do your job? 
24. What changes, if any, would you make to the Solutions program? 


25. (If this has not already been captured): How has the Solutions program increased 
the community college’s capacity to serve students? 
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